Error and Complexity

The learning curve model that I developed based on observations of how fast tasks are performed has yielded another valuable insight. The difference between expected performance of a task and actual performance over time, will tend to spike and then fall off gradually, which is likely to confound typically linear (and often overly-optimistic) approaches to scheduling its completion. Furthermore, the timing and amount of the spike will vary with the complexity of the task, which may not even be explicitly factored into expectations. The math shows that the timing and amount of this spike in what could be considered "error" is theoretically predictable.

In "Units of Completion," I suggested that the complexity of a task could be assessed in terms of a number of units that are simultaneously performed during the task, and which define how closely we can measure its completion. I've taken this another step, by using the concept of a unit to identify the highest meaningful efficiency that could be used to establish ideal expectations.

For example, if my task is to edit a page with 500 words, then the highest completion I could reliably measure is 499 words (500 minus one), which is 499/500 or 99.8% of the total. That fraction is also the highest meaningful efficiency, which translates into an expectation of editing 499 words in the best-case time. With average editing ability, I would have an efficiency of 50% instead of 99.8%; so during the best-case time, I will have only edited 50% of the total, or 250 words. If I'm responsible for meeting a schedule based on 99.8% efficiency, at the end of the best-case time I will be behind by 249 words (499 minus 250), or 49.8%, which is my error at that time, and it will take nine times that long to reach zero error. A manager tracking my progress up to the best-case time would see an even worse picture, because my error would reach a peak of 67% when I was at just 40% of the best-case time. Ideally, of course, the manager should plan for the actual time to achieve zero error, and not care what happens until then.

Editing ten pages instead of one could be treated as a single task, and the same fractions would simply apply to the larger number of words, with the minimum allowable error at the end now being ten words instead of one. If, however, this error was still kept at one word, then the highest completion (and the highest meaningful efficiency) would increase to 99.98% (or 4999/5000 words) and have significant side-effects which might together be considered a major degradation in performance. For one, the manager would now need to allow more than 12 times the new best-case time (which accounts for all ten pages) for me to reach zero error. My maximum error would increase to nearly 74%, occurring at 32% of the best-case time; and at the best-case time, my error would be slightly higher, at 50.0%.

My actual efficiency in editing pages is higher than the average, more like 70% than 50%, which would decrease the time and value of the maximum error, as well as the amount of time to reach zero error in each case. There would still appear to be a degrading effect on performance as the complexity increased (for example, the maximum error would have increased from 54% to 62%), and that could still raise an unnecessary "red flag" by a manager who was looking too closely and didn't expect it.

The real world is certainly messier than this theoretical discussion might imply. As I described in "Units of Completion," a lot depends on whether the task you think you're evaluating is one of these idealized pure tasks, a parallel combination of pure tasks, or a sequence of pure tasks. Since my analysis is based on actual observations (as are the other models I've developed), the behaviors I've identified are potentially observable in actual situations, and are therefore subject to test. They suggest a reasonable set of explanations for what may be unresolved or even unrecognized issues in actual applications, which is why I've brought them up.

One such issue, which I alluded to and can foresee, is an increase in waste: wasted time, wasted effort, and wasted physical resources. For example, a coordinated "task" such as a major industrial or government project might be terminated because of commitment to unrealistic planning goals that could not be met, and the waste of discontinuing it would be added to the loss of opportunity for meeting the needs it was intended to address. Spikes in what I've called "error" might result in the waste of resources to correct problems that don't exist, which rings true as a consequence of too much complexity. If more realistic schedules are impractical, either because they demand resources that aren't available, or because of competition with others who do not acknowledge their necessity, then the gains of previous effort should be preserved as much as possible until a new and more effective task -- or set of tasks -- can be devised. If preservation cannot be done, waste seems inevitable, and the ultimate objectives of the task are too important to abandon, then cooperation (rather than competition) may be needed between multiple entities who can together address the impediments to success.

Dangerous Optimism

I've seen it so often now that I can put numbers to it, and it applies to both individuals and organizations. A goal is set based on an optimistic judgement of what's involved, and one of the following results: (1) the goal isn't met; (2) the quality of what's done ends up being a fraction of what was intended; or (3) the goal is met, but someone else pays a large part of the price for it. Typically, the effort and resources required to accomplish any given task is two to three times these optimistic estimates, a factor that determines either the diminishing of quality or the "externalized cost" incurred.

For those who are successful at externalizing costs, it can appear that they are up to three times more efficient than others who incur more of the costs. Where competition for resources depends on achievement of more aggressive goals and on low internalized costs, such as in the world economy, everyone involved eventually must externalize as much as possible just to survive, and waste grows exponentially.

The poorest people are often those who have been forced to bear the costs pushed off by more "successful" people. As the waste grows, it overwhelms the ability of people to deal with it, and the number of poor and dying people grows with it. The world's combined waste is now nearly 60% of what the rest of nature can annually recycle, and still growing. The consequences are becoming catastrophic; like a Ponzi scheme, if it keeps going then everyone loses (I estimate this to be when the waste nears 75%).

Obviously, the simplest solution is to try to make more realistic assessments of the resources involved in meeting our goals, and to put as at least as much effort in assessing the impacts of our actions on everyone and everything potentially affected by them. In the simplest case, we would triple our estimates of required resources. The extreme case would involve full accountability: all of us honestly studying and publicizing the full effects of what we do and plan to do. This would no doubt slow down "progress," but it would also make it less destructive. Any error would be in large part due to lack of knowledge and understanding, which could be reduced by targeted efforts to increase both (as science is dedicated to doing).

Such solutions are, on their face, extremely unrealistic. It is human nature to compete, and deception is easier (and cheaper) than improving performance as long as the other competitors can be kept ignorant until the prize is won. Even a single competitor willing to deceive will decrease any value to performance for everyone else. A compromise is to have judges, which in society may consist of unbiased entities such as governments and journalists. The success of many of today's societies is often determined by how successful these entities are in maintaining their objectivity and power to limit deception.

Governments have the larger role of ensuring that their citizens can meet their basic needs, which includes keeping waste below an amount that overwhelms people's ability to survive. Not all governments acknowledge this role, and factions within every country actively debate its validity, but it remains a role that must be assumed by someone (if not everyone) within a society for the society to function.

Ideally – and I would argue, no matter what – plans for every goal, by every individual and group, should provide for preserving this basic right for the people impacted by the actions taken to meet the goal. Those who believe in limited government should take this into account before they act on their belief, or they should accept responsibility for the harm caused by not meeting this fundamental responsibility of society. We can't enjoy the benefits of society without providing for its survival, and that means avoiding dangerously optimistic planning that increases the waste that society has to deal with.