Efficiency Revisited

In "Efficiency and Completion Time," I discussed how the amount of time it takes to complete a generic task typically takes longer than the minimum possible time. I've since explored the math behind that observation, and how it might vary across the world's population using insights described in "Changing People."

It turns out that what I called the "efficiency" of performing a task is equivalent to what we commonly know as a compounding interest rate on a credit card we're not paying off, where we are exponentially depleting the remaining amount of the task. If efficiency varies randomly from zero to one across the population, then the 68% of people in the central bulge of the bell curve will have efficiencies between about 0.4 and 0.6. That is, a typical person will accomplish between 40% and 60% of the remaining part of a task during a period of time that is the best case for accomplishing the entire task, which includes preparation and perfect luck. Very skilled people working on state-of-the-art projects, in my experience, are at the high end of "typical" efficiency, which is at the low end of the top 16% of the population that includes only one person who can achieve the task in the best-case time.

Realistically there is a limit to luck, which based on my testing experience is around 95%. This means that multiple attempts at a task are unlikely to all achieve more than 95% of it. Typical people take about 3 to 6 times as long as the best case to complete 95% of a task. Only 47 people on our whole planet could accomplish 95% of a task in the best-case time or less (notably, at the beginning of civilization this number was ten – at least two breeding couples – with this efficiency).

Of course, tasks are extremely variable, which would seem to make these kinds of comparisons meaningless. That variability is, however, covered by the definition of best-case time, which includes the influence of such things as technological aids, the education available to the population, and inherent complexity. Efficiency is explicitly tied to just what humans can do, which is subject to finitely limited physical and mental capabilities that influence the performance of all tasks, so the time to complete a task must be measured as the time that humans are involved in activities that support it, which includes education, skill development, and acquisition of resources.

It is also important to note that each person may not have the same efficiency for all tasks: for example, someone with 20% efficiency for one task may have 80% efficiency for another task; the bell curve simply represents the number of people who have given efficiencies. Tasks may also have common aspects that allow people to have about the same efficiency for multiple tasks, especially those that enable them to meet their basic needs in a natural environment.

In this context, our social infrastructure, which includes education, can be seen as a means of accelerating the effective completion level of common tasks to a point where further action on specific tasks takes a reasonable amount of time, especially those that affect the goals of the society – the most critical being its survival. This also involves preparing people to cooperate (another common task) so they can collectively benefit from each other's strengths (high efficiencies), and offset each other's weaknesses (low efficiencies) under changing conditions that force adaptation through performance of modified or additional tasks. With a random distribution of efficiencies across the world's population, it would take seven times the best time to prepare 99% of the population to do a common task in the best time; that factor, and the limit of time available (perhaps 13 hours per day) would determine how many common tasks could be prepared for.

One of the most common tasks in human history is our joint effort to dominate the natural world, which materially translates into maximizing our consumption of ecological resources (our ecological footprint). Our achievement of the task on a per-person basis has accelerated, with the best-case time since the beginning of civilization shrinking through radical changes from more than 220,000 years to around one year, and the appearance that we may have already achieved the task around 2011 and are now going backwards.

Complexity and Hubris

In the book "Immoderate Greatness," author William Ophuls describes how an organization like a civilization grows complex by creating chaos and degradation in its environment, eventually using up or rendering useless one or more resources that is critical to its survival. Ironically, the complexity works against it, because the people in the civilization are inherently unable to understand it well enough to predict its effects, and can't recognize the symptoms of impending decline. When the decline starts, chaos grows and overwhelms them. They then react inappropriately, and the civilization rapidly collapses. This concurs with other things I've read, along with observations of how governments and businesses tend to develop.

There are, of course, some people who recognize the symptoms and what they mean. Unfortunately, they're not in the majority, and certainly not in the majority of leadership positions. As our worldwide civilization rapidly approaches the threshold where nothing can save it, in part due to the consequences of global warming, the majority of those who know what's coming hold out hope that somehow enlightenment will spread or technology will triumph, just in time. The rest have already given up, and are focused on explaining what's happening while salvaging what will be left by developing alternative values and ways of living like what I suggested in Beyond Hope.

In Efficiency and Completion Time, I described how progress on "tasks" evolves over time and depends upon preparation, action, and luck. As a test engineer, it's been my job to help designers and manufacturers to determine how much of a task remains, where the task is the creation and deployment of a technological system that meets a set of expectations called "requirements." This part of the preparation phase can only be done after the first attempt at completing the task, and real-world conditions (including how people will use the system) can be applied to demonstrate what the system will actually do, along with what affects it. Requirements, more often than not, are very simplistic guesses which should be modified or supplemented based on experience that checks assumptions they were based on and reveals unintended consequences of their application. I say "should be," because my specialty is finding these oversights, which tend to comprise most of problems discovered after the second attempt at completing the task, which can be up to 25% of the total task. Unfortunately, managers and those who pay them typically only plan for no more than two attempts, using their best guess as to how much time and resources are needed, and compensating for luck by hiring the most experienced and capable people they can find.

This a good example from my experience of the linear thinking that Ophuls attributes to system failure. I've heard it explained away as "realistic," and "pragmatic" by people who swear they would do more "in a perfect world." Yet they are also typically people who don't have the "bandwidth" (read "limit to the rate they can process information") to handle explanations that can't be captured in single pages of bullet points. To be fair, all of us can keep only a handful of ideas in our head at a time (I've heard between 3 and 7); and the people who have the power to decide what others should do often have more than a handful of complex tasks of their own that they are expected to work on simultaneously. The more power they have, the less time they have to do any part of it, even if they're highly efficient and work every waking hour, so it's no wonder that more than two attempts at a task is considered a luxury, and that 80% completion is considered acceptable, with the rest – hopefully – undetected, explained away, or blamed on someone else (such as bad luck as "acts of God").

Given enough time, we will experience the consequences of ignoring details that we missed. These consequences pile up, especially since we're driven to accomplish more and more, and the consequences interact to create amplifying feedback loops. Eventually they can't be ignored; but as Ophuls points out, by that point it may be too late to fix the underlying problems before we're overwhelmed. To the extent that people recognize this as a valid threat, they might be inclined to limit what they do to the consequences they can adequately foresee (matching power to acceptable responsibility), but our culture – and arguably human nature – makes their taking action on it highly unlikely. Most likely, people like me who are good at identifying the problems that aren't obvious, and yet potentially the most destructive in the long term, will be vilified, shunned, or merely tolerated, especially if we're vocal about what we find and what needs to be done to fix it.

Ophuls considers what I call "graceful shutdown" all but impossible without an extremely unlikely shift in values to ones like those I've promoted. I'm obviously inclined to agree; but I've grown even less optimistic than he is that it will happen, given the forces we have collectively unleashed with the prime focus of accelerating global extinction rates. In short, having already lost hope for civilization, I'm now a hair's-breadth away from searching for the planetary equivalent of hospice.

Efficiency and Completion Time

For a long time I've been puzzled by why it seems to take much longer to do something than I first guess. I think I may now have an answer.

A typical task has three components: preparation, action, and luck. If we know exactly what to do, have the resources we need, and have no bad luck, then we can accomplish a task in a minimum amount of time. There is no preparation, and we will be 100% efficient in completing the task. If these conditions are not met, then in the same amount of time we will only accomplish a fraction of the task, with some of that time taken learning, acquiring resources, and dealing with the impact of more normal luck. In my experience, the first attempt at doing something within an ideal timeframe (what I call an "iteration") will result in the task being, at best, about 70% complete. If you're familiar with the normal probability (or "bell") curve, that's the area under the curve that's roughly within plus-or-minus one standard deviation of the mean.

Sometimes we don't even know that we haven't completed the task. With writing, for example, reading what I've written often uncovers problems with what I wrote. When the remaining amount of the task is identified, and if I have the opportunity to work on it, I may be able to knock out 70% of it on the second iteration, which began with the review that uncovered the remainder. This still leaves 9% of the original task left undone. In many situations, the 91% that I've accomplished may be good enough; for others, even that isn't acceptable.

On my third iteration, I will typically spend most of my time preparing: finding out what's left to do, and then getting what I need to do it. Once again, with typical luck, I'll at best complete 70% of what's left, driving the total up to more than 97%. In most situations where someone else decides what I'm working on and how long it should take, anything more than two iterations is a luxury (and I often have to take the second iteration out of my hide), with three iterations being the absolute maximum.

I am considerably worse at bowling than at writing and editing. I recently played five games following a 13 year hiatus. The first game, which counts as an iteration, was better than I expected: 68. From this starting point, which was no doubt the result of my previous experience, my efficiency averaged less than 2% (attempting the maximum score of 300, with values per game varying from -6.5% to 22.7%). If the model holds true and my efficiency doesn't change, I'll need to play at least 322 additional games to consistently score 299 points.

The amount of time involved in action (working directly on the task), what I call "effort," is simply the reciprocal of efficiency. My 70% best case corresponds to 1/0.7 = 1.43, or 43% more time than if I was working at 100%, while the productive part of my bowling amounts to more than 60 times what it would take to bowl a perfect game.

Interestingly, the actual time, in iterations, is closely approximated by a linear function of effort, whose coefficients vary with how much of the task we expect to achieve. For example, to achieve 99.7% of a task (plus-or-minus 3 standard deviations on the bell curve), the number of iterations is approximated by multiplying effort by 6 and subtracting 3. A more modest goal of 95.5% (2 standard deviations) takes 3 times effort minus 1.5 iterations. The lowest amount I've seen professionals accept is 80% (part of the so-called "eighty-twenty rule"), which interestingly is nearly as close as the iteration approximation gets to a pure multiple of effort: 1.5 times effort.

Both my major professions, test engineering and technical writing (the editing part), involve identifying the parts of tasks that have not been completed by other people. Based on that experience alone, I expect this model to apply to everyone, with efficiencies comparable to mine. Without in-depth scientific research to back it up, I can only propose it as an hypothesis, and explore some potential consequences and questions that derive from it, if it is true.

Education is an obvious area of exploration. Should education be redefined as a means of enabling people to perform multiple iterations of the components of tasks they will encounter elsewhere, so they can use their innate efficiencies to achieve acceptable starting points for those future tasks (much as my bowling games built on experience from years ago, which built on component tasks of walking and throwing)? Is efficiency innate, or can it be modified (and if so, is this task-dependent)?

How does this affect planning and execution of complex projects that have multiple dependent and independent tasks being completed by people with different efficiencies and access to resources? What are the implications for waste from such projects, at scales up to and including global civilization, especially on the survival of everyone and everything impacted by it?

Efficiency

I revamped my consumption model to take account of ecological footprint going back over 500 years, and was able to closely predict population growth over that period. Looking ahead, it pushed out by 20 years the amount of time we have before our population crashes. My goal was to reliably estimate the time in history when we each used the amount of resources we must use in the future to avoid losses in our population and the crash of other species’ populations. The answer, depending on how long the world reduces the annual rate of consumption before settling at a fixed amount of consumption per year, is some 300 years ago, at the start of the Industrial Revolution.

Put another way, the world must increase its efficiency of per capita resource use by a factor of about 100 to keep our population from falling after it peaks, and by a factor of several hundred to keep other species’ populations from crashing. Interestingly, our (western) economy wastes roughly 100 pounds (in production and packaging) for every pound of product used (not counting externalities such as gas emissions and energy use during transportation), implying that we would need to extract only those resources that we actually use while cutting out the other costs.