The signal is the truth. The noise is what distracts us from the truth. – Nate Silver

Do you know (in PSI or kPa) what a proper oil pressure reading is for your car? Or the correct voltage reading for your alternator? Or at what temperature your engine should be running?

I sure don’t and I’d wager that the vast majority of car owners don’t either. That’s why over time, measurement gauges on a car’s dashboard have been replaced (or augmented with) warning lights – something to let you know that there is a problem that requires your attention. (As an aside, my grandfather who was a mechanic for 50+ years used to refer to them as “idiot lights” – as if anyone who doesn’t readily know the answer to my questions above shouldn’t even be allowed to drive).

While it’s true that gauges provide more information (a range of possible values versus just a binary on/off signal), you really need to fully understand the capabilities and limits of the systems being measured to know what they’re actually telling you and – more importantly – if action is required based on the reading, which you would need to monitor constantly.

I have been in the supply chain planning game for over 30 years now and when I meet a new colleague for the first time, it’s not uncommon for him/her to lead with something like “Our forecast accuracy is 79.6%” (I suppose they must be looking for some sort of affirmation or praise).

What they end up getting is usually a confused look and about a thousand questions:

• Are you measuring at item/location level then aggregating or aggregating then measuring?
• What specific measurement are you using?
• Over what time frame?
• Are the errors biased in one direction or the other?
• Etc., etc., etc.

However, the most important questions (which take some time to come to the surface) are:

• How do you know if that’s good or bad?; and
• What do you do with that information?

When you multiply the number of items through the number of locations for a retailer, there are a lot of forecasts being produced across fast sellers, slow sellers, seasonal items, highly promoted items, products at the beginning/end of their lifecycle, not to mention myriad combinations of all of those attributes.

There’s nothing mathematically invalid about aggregating numbers and dividing them by each other to come up with a percentage using the full data set, but there’s nothing intrinsically useful about it either. Unless – and this will sound a bit cynical – your goal is to produce a number that makes it appear as though the process is working beautifully, even though it may be completely skewed by very high volume and stable selling items that require no effort to forecast well.

Most importantly, it doesn’t give a true picture of the highly varied landscape that is retail forecasting and it’s not actionable.

The forecasts that matter are the ones that directly affect the customer experience – for each specific item at each specific location. I’ve discussed this before, but when you’re looking at the individual granular forecasts that actually run the supply chain, you need to be accurate, not precise (those are two different things). It means moving away from comparing a single forecast to a single actual and calculating percentages to determine distance from absolute perfection (a completely unreasonable yardstick to measure against).

It’s really about recognizing that for any given item at any given store at any given time, there is a range expected forecast values that can be considered high quality, depending on the nature of the product, the timing of the demand and the additional causals that might be in play at the time.

Think of it this way: If your forecast for an item at a store is 4 units for the week and you actually sold 2, your “accuracy” is 50%, which sounds bad. But is it really a bad forecast? Is there something you could have done in advance that would have led you to forecast a value closer to 4? Now that you know you missed by 2 units, what specific actions can you take to make sure the next forecast is closer? What you end up doing in this scenario is chasing noise – and in retail forecasting there is a lot of it.

However, at this level of volume, it’s probably quite common to miss by 3 units in either direction in any given week, just as a result of uncontrollable randomness. In other words, there is an expected and acceptable “distance from perfect” that applies to every forecast. To make your forecast measurement truly actionable and useful, you should be measuring actuals against this range to make this determination. So, in our example above, if the actual is 4 and the common range of error is +/- 3 units, then any forecast between 1 and 7 units would be considered accurate when compared against the actual.

If you use a probabilistic forecasting approach, these acceptable ranges can be calculated with the forecast itself. If not, there would need to be some upfront analysis to assign one to each forecast, but the end result is that you have a way to find the needles in the haystack: instead of knowing which forecasts have error (pretty much all of them), you instead know which forecasts have unexpectedly high error. That’s actionable information that can be chased down and investigated.

With each individual forecast classified in this way (essentially an accurate/inaccurate “warning light”), you can also present a more meaningful gauge on your high level dashboard. Instead of a number representing “distance from unattainable perfection in the aggregate”, you can instead report on percentage of forecasts out of error tolerance.

 

A good forecaster is no smarter than everyone else, he merely has his ignorance better organized. – Anonymous

I’ve written on the topic of forecast performance measurement from many different angles, particularly in the context of forecasting sales at the point of consumption in retail.

Over the years, I’ve opined that:

• Forecast accuracy (in the traditional sense) is a useless measure
• Reasonableness is more important than accuracy, given that forecasts are, by their nature, forgiving planning elements
• The outsized importance placed on forecast accuracy in supply chain planning is a myth
• Accuracy and precision must be considered simultaneously
• Forecasts should be judged against what is a reasonable expectation for accuracy
• Forecasting at higher levels of aggregation to achieve higher levels of “accuracy” is a waste of time

After going back and re-reading all of that stuff, they are all really just different angles and approaches for delivering the message “popular methods of comparing forecasts and actuals may not be as useful as you think, especially in a retail context”.

But in all of this time there is one key aspect of forecast measurement that I have not addressed: forecast lags. In other words, which forecast (or forecasts) should you be comparing to the actual?

Assuming, for example, that you have a rolling 52 week forecasting process where forecasts and actuals are in weekly buckets, then for any given week, you would have 52 choices of forecasts to compare to a single actual. So which one(s) do you choose?

Let’s get the easy one out of the way first. Considering that the forecast is being used to drive the supply chain, the conventional wisdom is that the most important lag to capture for measurement  is the order lead time, when a firm commitment to purchase must be made based on the forecast. For example, if the lead time is 4 weeks, you’d capture the forecast for 4 weeks from now and measure its accuracy when the actual is posted 4 weeks later.

Nope. To all of that.

While it’s true that measuring the cumulative forecast error over the lead time can be useful for determining safety stock levels, it’s not very useful for measuring the performance of the forecasting process itself, for a couple of reasons:

1. It is a flagrant violation of demand planning principle. Nothing on the supply side of the equation (inventory levels, lead times, pack rounding, purchasing constraints, etc.) has anything to do with true demand. Customers want the products they want, where they want them and when they want them at a price they’re willing to pay, period. The amount of time it happens to take to get from the point of origin to a customer accessible location is completely immaterial to the customer.
2. A demand planner’s job is to manage the entire continuum of forecasts over the forecast horizon. If they know about something that will affect demand at any point (or at all points) over the next 52 weeks, the forecasts should be amended accordingly.

Suppose that you’re a demand planner who manages the following item/location. The black line is 3 years’ worth of demand history and a weekly baseline forecast is calculated for the next 52 weeks.

Because you’re a very good demand planner who keeps tabs on the drivers of demand for this product, you know that:

• The warm weather that drives the demand pattern for this item/location has arrived early and it looks like it’s going to stay that way between now and when the season was originally expected to start.
• There are 2 one week price promotions coming up that have just been signed off and all of the pertinent details (particularly timing and discount) are known.
• For the last 3 years, there have been 3 similar products to this one being offered at this location. A decision has just been made to broaden the assortment with 2 additional similar products half way through the selling season.

On that basis, I have 2 questions:

1. How does the baseline forecast need to change in order to incorporate this new information?
2. How would your answer to question 1 change if you also knew that the order-to-delivery lead time for this item/location was 1 week? 2 weeks? 12 weeks?

Hint: Because it was established at the outset that “you’re a very good demand planner who keeps tabs on the drivers of demand for this product”, the answer to question 2 is: “Not at all.”

So if measuring forecast error at the lead time isn’t the right way to go, then what lag(s) should be captured for measurement?

As with all things forecasting related, there is no definitive answer to this question. But as a matter of principle, the lags chosen to measure the performance of a demand planning process should based on when facts become “knowable” that could affect future demand and would prompt a demand planner to “grab the stick” and override a baseline forecast modeled based on historical patterns.

In some cases, upstream processes that create or shape demand can provide very specific input to the forecasting process.

For example, it’s common for retailers to have promotional planning processes with specific milestones, for example:

• Product selection and price discounts are decided 12 weeks out
• Final design of media to support the ad is decided 8 weeks out
• Last minute adds, deletes and switches are finalized 3 weeks out

At each of those milestones, decisions can be made that might impact a demand planner’s expectation of demand for the promotion, so in this case, it would be valuable to store forecasts at lags 3, 8 and 12. Similar milestone schedules generally exist for assortment decisions as well.

In other cases, what’s “knowable” to the demand planner can be subject to judgment. For example, if actuals come in higher than forecast for 3 weeks in a row, is that a trend change or a blip? How about 4 weeks in a row?

Lags that are closer in time (say 0 through 4) are often useful in this regard, as they can show error trends forming while they are still fresh.

Unless tied to a demand shaping process with specific milestones as described above, long term lags are virtually useless. Reviewing actuals posted over the weekend and comparing it to a forecast for that week that was created 6 months ago might be an interesting academic exercise, but it’s a complete waste of time otherwise.

The key of measuring is to inform so as to improve the process over the long term.

With the right tools and mindset, today’s “I wish I knew that ahead of time” turns into tomorrow’s knowable information.