Forecast value added against a naive baseline

Today let's understand forecast value added, the number that finally answers a question every forecasting team eventually has to face: did the model actually earn the effort it took to build, or would a forecast anyone could write in one line of code have done just as well?

Here is SKU-05, one private-label item from a regional grocery chain's 18-SKU portfolio. The chain has 60 months of unit sales for it, and the forecasting team fits an ETS model, error-trend-seasonal exponential smoothing that picks its own level, trend and seasonal pattern from the data, on the first 48 of those months, then checks the forecast against the final 12.

Look at how flat the ETS forecast line sits across the final 12 months, while the actual sales swing well above it for most of that stretch. Over those 12 months, the ETS model's forecast missed actual sales by an average of 65.9 units a month. A seasonal naive forecast, the kind that just repeats whatever this SKU sold the same month last year, missed by only 36.8 units. So the model that took real engineering effort to build did worse than a forecast that takes one line of code. That gap is what forecast value added measures, and for SKU-05 it comes out negative.

The naive and seasonal naive forecasts

An error number on its own tells you almost nothing. 65.9 units a month, is that good or bad? It depends entirely on what you compare it against, and the fairest comparison is a benchmark that costs nothing to produce.

The simplest such benchmark is the naive forecast: repeat whatever the series did most recently, for every future period. The seasonal naive forecast does one thing differently. It repeats whatever happened at the same point in the last full cycle, in a monthly series, the same month a year back. Monthly retail sales almost always carry a yearly cycle, more sales in some months than others, repeating every 12 months, so a plain naive forecast would miss that shape completely, while seasonal naive captures it for free.

Build all 18 of the chain's SKUs the way the forecasting team actually sees them: 60 months of history each, with the first 48 months set aside to fit a model on and the final 12 held back to check it against. That held-back stretch is called the holdout: the real sales a forecast is finally judged against.

SKU-01's holdout starts at month 49. The naive forecast just repeats month 48's value, 271 units, for every one of the next four months. The seasonal naive forecast repeats months 37 to 40, the same four months one year earlier: 338, 315, 314 and 352, which already sit much closer to where actual sales landed.

Averaged over those four months, the naive forecast misses by 96.5 units. The seasonal naive forecast misses by only 37.75. That's the whole reason seasonal naive, and not plain naive, is the right minimal yardstick for a series shaped like this one.