Phil Tetlock spent almost two decades determining people’s ability to forecast specific events – things like elections and the outcomes of potential geopolitical decisions. Unfortunately, the results were not impressive. Most people were about as accurate as a well-fed, dart-throwing chimpanzee.
There were a small number of notable exceptions. A small group of people consistently provided quite accurate forecasts – people he aptly named “super-forecasters”.
In a subsequent forecasting tournament organized by the Intelligence Advanced Research Projects Activity (IARPA) – a focused branch of the United States intelligence community, the super-forecasters trounced teams of professors and “forecasting experts” by wide margins.
What made the super-forecasters so super?
It wasn’t intelligence or that they had more experience than others. In fact, in many cases, they were mostly amateurs yet they outperformed the CIA’s best and brightest (who also had the advantage of years of experience and classified information). Armed with only Google, the super-forecasters beat the CIA, on average, by 30%.
What made them great at being right was they were great at being wrong!
The difference in their ability to forecast was simple, yet crucial. The super-forecasters changed their minds – a lot.
Not a huge, 180-degree shift, but subtle revisions to their predictions as they learned new information. As an example, one of the consistently top super-forecasters would routinely change his mind at least a dozen times on a prediction and, sometimes, as often as forty or fifty times.
Importantly, most viewed a revised forecast based on new information not as changing an initially wrong forecast but rather as updating it. Turns out, updating is the secret to being a great, or super, forecaster.
The concept of updating is important in Flowcasting as well.
As loyal Flowcasters know, an important component of the process is the sharing of what we call a supplier schedule – that is, a projection, by item and delivery location of how many units are needed to ship over a long time horizon (typically 52+ weeks).
If the schedule indicates that, 39 weeks from now, the supplier will need to ship 10440 units of a product to a location, what’s the chance that this projection (i.e., forecast) is perfectly accurate? Pretty low, right? And it doesn’t need to be – it just needs to be reasonable.
A week later, and guess what? The supplier schedule has been re-calculated and updated to indicate that 10400 units are needed to ship in that particular week (perhaps even on a different day). The updated forecast is more right than the previous one. This process of minor revisions continues as the projections are updated until it’s actually time to ship (i.e., when the planned shipment has reached the agreed-upon order release horizon).
In retail, supplier scheduling is the super-forecaster for suppliers – recalibrating and subtly updating the forward looking projections, based on the latest information until…
Wrong becomes right.