Using Averages in Your Business Data Analysis? You May Be Overlooking Risk.

By Sean Brooks

Have you ever wondered why your sales forecasts are off, you overshoot your budget estimates, or your projects miss critical deadlines? You believe that you’ve conducted thorough data analyses and yet the frustration of inaccurate estimates continues to haunt you. So what’s the problem? Where have you gone wrong?

You may be impairing your plans from the beginning by building them with numbers that represent “best guesses” in place of uncertain inputs. In other words, you’ve put one number into each spreadsheet cell that actually represents an uncertain, future quantity. Consider the following example:

A major U.S. bank has invested resources into innovative analytics techniques and data-intensive modeling to assess and reduce credit risk and to maximize earnings. It also has been amending the understated book value of its bad debt expense after having experienced a dramatic increase in loan defaults.

The bank had been following GAAP, which only requires that bad debt be entered as a single number. Despite their sophisticated scenario models, bank leaders failed to account appropriately for uncertainty, or the inevitable variation in bad debt expense influenced by factors such as collateral value, unemployment, and borrowers’ personal finances.

Over time, this practice resulted in overstated asset quality and major accounting inaccuracies. Damaging adjustments to correct previously overstated earnings could raise questions about the bank’s earnings quality and management ethics.

So what’s going on here?

This bank has fallen victim to the Flaw of Averages, a term coined by Stanford professor Sam Savage, who has articulated that plans based on average assumptions are wrong on average.

Savage explains that the Flaw of Averages is a series of mistakes that most of us make consistently when we substitute single numbers, often expressed as average values, into a model or any data analysis tool when there is uncertainty at play. In the example above, the bank may have averaged its historical bad debt and projected it into the future.

While a bank accounting example may not resonate with everyone, we’ve all seen the impact of this problem on an even larger scale before. Think back to pre-2008 when many people believed that the value of real estate and 401k investments would continue to increase because, on average, returns had been positive. In fact, the average value of anything is far from “most likely,” as it represents only one possible value in a much wider distribution, or range of possible outcomes.

Unfortunately, despite the painful and prolonged sting of the Great Recession, this concept is still not widely understood and practiced. The good news is that there are ways to avoid making this mistake in your business decisions.

Why should you care?

Many experienced executives and decision makers use single-number inputs in formal planning exercises as a standard practice, overlooking the opportunity to use distributions because they can be challenging to understand, communicate, and act on.

Single-number estimates are easy to understand and work with but almost always result in flawed analyses and poor business decision making when used as proxies for distributions. Just as an earthquake can topple a building with an insufficient foundation, unexpected occurrences can flatten plans built on analyses derived from static, single numbers.

Failure to account for uncertainty has serious consequences and may explain why forecasts, budgets, and project timelines end up being so off the mark. This one-dimensional approach is overly simplistic and downright dangerous for executives and managers whose decisions have major financial impacts.

What can you do about this now?

Fortunately, there are established techniques and accessible tools to harness the uncertainties in forecasts instead of looking the other way or settling for an easier (but flawed) alternative.

Monte Carlo simulation analyses provide experimental forecasts of the consequences of decisions in a probabilistic way. This technique packs a big punch by modeling thousands of single numbers into one spreadsheet cell as a distribution of inputs. You can adjust multiple parameter inputs, including distributions, in order to analyze sensitivity (e.g., how much money should we expect to budget for different levels of project risk?) and to test various expected outcomes (e.g., what are the probabilities that demand for our new product will exceed our capacity to produce it if we do nothing or if we increase capacity?).

Simulation enhances your business planning with interactive, dynamic calculations that will equip you to begin looking for answers to even more interesting questions: “How wide are the variations in an expected outcome, and how frequently should I expect them to occur? What are the root causes of these variations, and what can I do to reduce their impacts?”

Don’t feel helpless by an inability to understand and model uncertainty in next quarter’s sales or next year’s expenses.  Now you can perform calculations on uncertainties instead of on numbers only. You can model and manage uncertainty and improve your risk management and communication.

Final Verdict

A problem-solving approach by which you explicitly model uncertainty results in better decisions. Optimization and simulation analytic tools were not widely available until more recently. Even now, many business leaders and decision makers are not familiar enough with these techniques and so they don’t use them. But it’s time to start using them—or to outsource to advisors who can help—if you’re interested in improving the quality of your analyses and in making better decisions.

Professor Savage has prescribed the following approach: “Rather than ‘Give me a number for my report,’ what every executive should be saying is ‘Give me a distribution for my simulation.’”

5 Tips for Security Data Analysis