It ain’t over ’til the fat lady sings. – Modern Proverb

When is it too late to update a forecast?

Here’s a theoretical scenario. You’re a retailer who sells barbecue charcoal. The July 4th is approaching and a large spike in sales is predicted for that week.

Time marches on and now you’re at the beginning of the week in which the holiday is going to happen. For a large swath of the country, a large storm front is passing through and there’s no way in hell that people will be out barbecuing in their usual numbers.

Remember, the holiday is only a few days away now. Chances are that the stores have already received (or have en route) a large amount of charcoal based on the forecast that was in force when outbound shipments were being committed to the stores.

So, we’re already within the week of the forecasted event and most (if not all) of the product has already been shipped to support a sales forecast that is way too high. Nothing can be done at this point to change that outcome.

So changing the forecast to reflect the expected downturn in sales is basically pointless, right?

Au contraire.

When the entire supply chain is linked to the sales forecast at the store shelf, then the purpose of the forecast goes far beyond just replenishing the store.

The store sales forecast drives the store’s replenishment needs and the store replenishment needs drive the DC’s replenishment needs, and so on. All of this happens on a continuum that really has nothing to do with what’s already been committed and what hasn’t.

If your sales forecast for charcoal in the affected stores is 5,000 units over the next 5 days, but you know with a pretty high degree of certainty that you will only sell about 2,000 because of the weather, then why would you delay the process of realigning the entire supply chain to this new reality by several days just because you can’t affect the immediate outcome in the stores right now?

The point here is that while the supply chain is constrained, the sales forecast that drives it is not. It may not be possible for a forecast update to change orders that are already en route, but it is always possible to change the next planned order based on the new reality. In that way, you already have a plan in place that is starting to get you out of trouble before the impact of the problem has even fully materialized. In other words, bad news early is better than bad news late.

If you have information that you think will materially impact sales, then the only time it’s too late to update the forecast is after it’s already happened.

Never compare your inside with somebody else’s outside – Hugh Macleod

I’m aware that this topic has been covered ad nauseum, but first a brief word on the subject of benchmarking your forecast accuracy against competitors or industry peers:

Don’t.

Does any company in the world have the exact same product mix that you do? The same market presence? The same merchandising and promotional strategies?

If your answer to all three of the above questions is ‘yes’, then you have a lot more to worry about than your forecast accuracy.

For the rest of you, you’re probably wondering to yourself: “How do I know if we’re doing a good job of forecasting?”

Should you measure MAPE? MAD/Mean? Weighted MAPE? Symmetric MAPE? Comparison to a naïve method? Should you be using different methods depending on volume?

Yes! Wait, no! Okay, maybe…

The problem here is that if you’re looking for some arithmetic equation to definitively tell you whether or not your forecasting process is working, you’re tilting at windmills.

It’s easy to measure on time performance: Either the shipment arrived on time or it didn’t. In cases where it didn’t, you can pinpoint where the failure occurred.

It’s easy to measure inventory record accuracy: Either the physical count matches the computer record or it doesn’t. In cases where it doesn’t, the number of variables that can contribute to the error is limited.

In both of the above cases (and most other supply chain performance metrics), near-perfection is an achievable goal if you have the resources and motivation to attack the underlying problems. You can always rank your performance in terms of ‘closeness to 100%’.

Demand forecast accuracy is an entirely different animal. Demand is a function of human behaviour (which is often, but not always rational), weather, the actions of your competitors and completely unforeseen events whose impact on demand only makes sense through hindsight.

So is measuring forecast accuracy pointless?

Of course not, so long as you acknowledge that the goal is continuous improvement, not ‘closeness to 100%’ or ‘at least as good as our competitors’. And, for God’s sake, don’t rank and reward (or punish) your demand planners based solely on how accurate their forecasts are!

Always remember that a forecast is the result of a process and that people’s performance and accountability should be measured on things that they can directly control.

Also, reasonableness is what you’re ultimately striving for, not some arbitrary accuracy measurement. As a case in point, item/store level demand can be extremely low for the majority of items in any retail enterprise. If a forecast is 1 unit for a week and you sell 3, that’s a 67% error rate – but was it really a bad forecast?

A much better way to think of forecast performance is in terms of tolerance. For products that sell 10-20 units per year at a store, a MAPE of 70% might be quite tolerable. But for items that sell 100-200 units per week a MAPE of 30% might be unacceptable.

Start by just setting a sliding scale based on volume, using whatever level of error you’re currently achieving for each volume level as a benchmark ‘tolerance’. It doesn’t matter so much where you set the tolerances – it only matters that the tolerances you set are grounded in reasonableness.

Your overall forecast performance is a simple ratio: Number of Forecasts Outside Tolerance / Number of Forecasts Produced * 100%.

Whenever your error rate exceeds tolerance (for that item’s volume level), you need to figure out what caused the error to be abnormally high and, more importantly, if any change to the process could have prevented that error from occurring.

Perhaps your promotional forecasts are always biased to the high side. Does everyone involved in the process understand that the goal is to rationally predict the demand, not provide an aspirational target?

Perhaps demand at a particular store is skyrocketing for a group of items because a nearby competitor closed up shop. Do you have a process whereby the people in the field can communicate this information to the demand planning group?

Perhaps sales of a seasonal line is in the doldrums because spring is breaking late in a large swath of the country. Have your seasonal demand planners been watching the Weather Channel?

Not every out of tolerance forecast result has an explanation. And not every out of tolerance forecast with an explanation has a remedy.

But some do.

Working your errors in this fashion is where demand insight comes from. Over time, your forecasts out of tolerance will drop and your understanding of demand drivers will increase. Then you can tighten the tolerances a little and start the cycle again.