Choosing how often to refit a forecast model
Today let's understand how often a forecast model should be refit, and what goes wrong on either side of the right answer.
Highgate Paper Co. is a wholesale paper products supplier. Its retail stationery customers place orders every week, and Highgate forecasts next week's order volume to plan production and shipping. Here are its last 100 weeks of orders, in cases a week. For the first 60 weeks, demand holds steady around 300 cases. Then, at week 61, a regional retail chain starts placing standing weekly orders, and the level steps up to 390 cases and stays there.
Highgate's forecast comes from a model, and like any forecasting model, it has to be refit now and then: re-estimated on the latest orders. Refit it too often, and you risk chasing noise instead of a real pattern. Refit it too rarely, and you risk missing that the level actually moved, the way it does at week 61 above. This lesson works out, from Highgate's own 100 weeks, how often is actually often enough.
The two ways a refit schedule goes wrong
Refit means re-estimating a forecasting model's parameters on new data. For Highgate's order forecasts, that means recomputing whatever the model uses to predict next week's orders, using the most recent weeks of actual orders.
A refit schedule can fail two different ways. Refit too often, and ordinary week-to-week noise pushes the forecast around, even on weeks when nothing about the real level of demand has changed. Refit too rarely, and the forecast keeps repeating an old level long after the real level has moved, the way Highgate's does at week 61.
Both failures cost real forecast accuracy, and the cost can be measured directly in Highgate's own numbers.
A model that refits every week
Start with the model Highgate actually uses: the trailing 8-week average. Each week, it looks back over the last 8 weeks of orders, averages them, and uses that average as the forecast for the next week. Refitting this model just means recomputing that average once a new week of orders comes in.
Build the full 100-week series from the same fixed formula every time, then compute the trailing 8-week average for every week it can be computed, week 8 onward.
Look at weeks 55 through 61. For most of those weeks the forecast sits in the 295 to 299 range, close to the true pre-shift level of 300. By week 61, the first post-shift week, it has barely moved, 304.75, because the 8-week window behind it, weeks 54 through 61, still mostly holds pre-shift orders. The forecast looks accurate here, but only because nothing has changed yet as far as it can tell.
How much weekly refitting moves when nothing has changed
Weekly refitting sounds safe: new data in, forecast updated, repeat. But even across the 39 weeks from 22 to 60, where the true level of demand never moves from 300, the trailing average itself keeps moving a little every week. That happens because it is built from 8 noisy weeks, and no 8-week window of noisy orders averages to exactly the same number as the window before it.
Measure that week-to-week movement directly, over weeks 22 to 60.
The forecast never moves by more than 3.5 cases from one week to the next in this stretch, and it moves by 1.4 cases on average. That is the cost of refitting every week: even with nothing really changing, the forecast still wobbles by a case or two, purely from which 8 weeks of noise happen to sit in the window. Call this the noise floor. Every refit schedule has to live with some version of it.
A model that is never refit again
Now go to the opposite extreme. Fit the trailing average once, on weeks 45 through 60, and then freeze it: never recompute it again, no matter what orders come in afterward.
To compare the frozen model against weekly refitting, measure each one's mean absolute error: for each week in the stretch being scored, take the forecast's miss from the actual orders, drop its sign so a miss of 10 counts the same whether it was too high or too low, then average those misses. The result stays in cases a week, the same unit as the orders themselves.
The frozen forecast is 297.81 cases, a perfectly reasonable number for the stable weeks it was fit on. See what that fixed number costs against weekly refitting over weeks 61 to 100, the 40 weeks after the real step to 390.
Weekly refitting's mean absolute error over those 40 weeks is 17.53 cases. The frozen model's is 91.36, more than 5 times worse. The frozen forecast has no way to reflect that the chain started ordering at week 61, so it just keeps repeating 297.81 forever, while actual orders sit near 390. The gap between the flat line and the real data in the chart above is the entire cost of never refitting again.
Quick check: the two failure modes
Weekly refitting moved the forecast by at most 3.5 cases during the 39 weeks when nothing changed, and cost a mean absolute error of 17.53 over the 40 weeks after the real step. The frozen model never moved at all, and cost 91.36 over that same stretch. What does this pair of numbers actually show?
A fixed, in-between schedule is still a guess
Both extremes have now been measured: refit every week, or never again. A compromise seems obvious. Refit every 8 weeks instead, using the trailing average computed at that moment, and hold that number fixed until the next scheduled refit.
24.22 lands between weekly refitting's 17.53 and freezing's 91.36, which looks like a reasonable middle ground, until you see where that error actually comes from. The first 8 weeks right after the shift alone cost a mean absolute error of 68.88, almost as bad as freezing completely. That is the lag built into any fixed schedule: whatever just changed has to wait out the rest of the current 8-week block before the next scheduled refit even looks at it. A fixed interval cannot tell a quiet week apart from the one week that actually matters.
How to tell a real change from noise
The trailing average's own week-to-week change stays under 3.5 cases through the 39 stable weeks from 22 to 60. Extend that same measurement past the real shift at week 61, and watch what happens to it.
Through week 60, the week-to-week change in the forecast stays inside that same narrow band, never past 3.5 in either direction. Then, for seven straight weeks, 62 through 68, it jumps to between 10.75 and 13.12, more than 3 times anything seen in the stable weeks, as the real step from 300 to 390 works its way through the 8-week window one week at a time. These two regimes do not overlap at all. That gap, between how much the forecast moves on ordinary noise and how much it moves when the real level actually changes, is exactly the signal a refit schedule should be using.
Turning that signal into a refit rule
The trailing average's week-to-week change never passes 3.5 cases while nothing real is happening, but jumps past 9 once a real change is working through the window. That is a clean separation, so set a refit trigger just above the noise ceiling. Keep using whatever level is currently adopted as the forecast, and only refit, adopting the newly computed trailing average as the forecast going forward, when it differs from the level currently in use by more than 4 cases.
18 refits, against 92 for weekly refitting, land at almost the same accuracy: a mean absolute error of 17.55 against weekly's 17.53. That is a fifth of the refits for essentially the same result.
Honestly, not every one of those 18 refits was actually catching a real change. Only 8 of them, the last line of the output above, fall inside the shift itself, weeks 61 through 68. The other 10 happened during otherwise quiet weeks, where ordinary noise alone pushed the trailing average just past the 4-case threshold. That is the price of setting the threshold this close to the noise ceiling: it occasionally fires on noise, but it still refits far less often than a weekly schedule while matching its accuracy.
What the threshold rule misses, and the backstop for it
A trigger built from how fast the trailing average normally moves has one honest gap. A real change that moves the level slowly enough never pushes the week-to-week change past 4 in a single week, and a pure threshold rule would never refit for it at all, no matter how far the level eventually drifts.
The fix is a backstop on top of the threshold, not instead of it.
Refit the moment the threshold fires. But also set a maximum interval, for example 13 weeks, and refit anyway once that many weeks have passed with no trigger, even if nothing has crossed the threshold. A fast change gets caught by the threshold. A slow one still gets caught, just later, by the backstop.
Quick check: choosing a cadence policy
Weekly refitting costs 92 refits for a mean absolute error of 17.53. Refitting every 8 weeks costs 12 refits for 24.22. Never refitting again costs 0 refits for 91.36. And refitting only when the level moves by more than 4 costs 18 refits for 17.55, plus a maximum-interval backstop for drift too slow to trigger it. Which approach should Highgate actually adopt?
Your turn: refit or hold?
Highgate's threshold was 4 cases: refit whenever the trailing average moves by more than that from one week to the next. Suppose another series inside Highgate shows these week-to-week changes in its own trailing average: 1.1, -2.4, 3.0, 9.8, -0.6, 11.2. Using the same rule, find which of these six weeks call for a refit.
Show answer
# Flag every week where the change's absolute size exceeds the threshold
flags <- abs(changes) > threshold
which(flags)
#> [1] 4 6References
- Forecasting: Principles and Practice, 3rd edition - Hyndman, R.J. and Athanasopoulos, G. Chapters on exponential smoothing and time-series cross-validation.
- Designing Machine Learning Systems - Huyen, C. (2022), O'Reilly. Chapter on continual learning and retraining triggers versus fixed schedules.
- Hidden Technical Debt in Machine Learning Systems - Sculley, D. and collaborators (2015), NeurIPS.
- A Survey on Concept Drift Adaptation - Gama, J., Zliobaite, I., Bifet, A., Pechenizkiy, M., and Bouchachia, A. (2014), ACM Computing Surveys, 46(4).
What you can do now
By the end of this lesson, you can set a forecast model's refit cadence from its own measured noise, instead of from habit.
Weekly refitting cost a mean absolute error of 17.53 over the 40 weeks after Highgate's real shift, against 91.36 for never refitting again and 24.22 for a fixed every-8-week compromise that still lagged badly right after the shift. Measuring how far the forecast normally moves on pure noise, 3.5 cases at most, against how far it moves on a real change, past 9, gave a refit trigger that matched weekly refitting's accuracy, 17.55, using a fifth of the refits weekly refitting needed.
That trigger is not perfect on its own: a slow enough drift could still slip past it, which is why it runs alongside a maximum-interval backstop rather than alone.