Rolling-origin backtests without leakage

Today let's understand how to test a forecasting method properly, and the sneaky ways that test can make it look better than it is without you ever noticing.

Lumen & Co is a small online retailer, and its best-selling product is a desk lamp. Below are its weekly unit sales for the last 104 weeks, two full years.

Sales start around 40 units a week and climb to around 140, with the usual week-to-week bounce along the way. Over the first year that averages out to about 64 units a week. By the second year it has risen to about 115.

Suppose you build a forecasting method for this series, and you want to know how accurate it really is before you trust it with real reorder decisions. That rising line above is the only data this lesson uses. Everything from here on asks one question about it: can you trust a backtest run on it, or can the backtest quietly overstate how good that method really is?

Why an ordinary shuffled fold is itself a leak on a time series

If you have tested a model before, you have probably used cross-validation: split the rows into a handful of folds, hold one fold out as a test set, train on the rest, and repeat until every fold has had a turn as the test set. The rows land in their folds at random, so each fold is just some random slice of the whole dataset.

That works fine when the rows do not have an order to them. But Lumen's 104 weeks are not interchangeable rows. Week 30 comes before week 50, and a forecast for week 30 is only honest if it was built without ever touching week 50.

Here is what a random fold does to that order. Suppose you split Lumen's 104 weeks into a random 80% training set and a 20% test set, and week 30 happens to land in the test set, the week you are trying to forecast.

week 30            -> test
weeks 31 through 40 -> all land in the training set

Ten weeks that come right after week 30 on the calendar, weeks that have not even happened yet relative to week 30, end up training the very model being asked to forecast it. The model gets to peek at the future before it makes its guess, and that is the simplest leak there is.

So the fix is not a better random split. It is to stop assigning weeks to folds at random altogether, and build the split around the calendar instead.

Rolling-origin cross-validation: the origin, the window and the horizon

The fix is called rolling-origin cross-validation, and it rests on three ideas.

The origin is the last week you are allowed to train on. Everything up to and including the origin is fair game. Everything after it is the future, as far as that forecast is concerned.

The horizon, written h, is how many weeks past the origin you forecast. Lumen reorders stock every 4 weeks, so h = 4 throughout this lesson: an origin at week 52 forecasts weeks 53 to 56, nothing further out.

The training window expands as you go: it always starts at week 1 and runs to the origin, so it grows longer every time you move the origin forward. And rolling just means repeating this at several origins instead of one, moving the origin forward each time and scoring a fresh forecast.

Here is what that looks like at three of Lumen's origins.

Notice what never happens in this table. No training window ever reaches past its own origin. That single rule is what a shuffled fold breaks and rolling-origin cross-validation protects.

Building a correct rolling-origin backtest in R

Let's build this for real on Lumen's data and see how accurate the method actually is.

The model is simple on purpose: forecast sales from the week number (to catch the rising trend) and a trailing 4-week average (to catch the recent level). The trailing average at week i is just the mean of the 4 weeks strictly before it, weeks i-4 to i-1, so it never needs a week that has not happened yet.

Lumen rolls the origin forward every 4 weeks, from week 52 to week 100, which gives 13 origins in total. At each one, the model trains only on weeks up to that origin, then forecasts the next 4 weeks one at a time, feeding each week's own forecast forward to build the trailing average for the week after it. That last part matters: using the model's own forecast instead of peeking at the real future value for weeks 2, 3 and 4 of the horizon is itself part of keeping this backtest honest, which is why it gets its own line in the code below.

rmse_by_origin is one accuracy number per origin, and it swings from 2.02 up to 11.76 depending on which 4 weeks got forecast. Some 4-week stretches are just easier to call than others.

The last line pools every one of the 52 forecast errors, 13 origins times 4 weeks each, into a single number: RMSE, root mean squared error. It squares each error so a big miss counts more than a small one, averages the squares, then takes a square root to bring the units back to sales, not squared sales. That gives 8.48. This is the honest number: the real accuracy of this method, measured the way it would actually be used. Every leak in this lesson is a different way of making that 8.48 look better than it is, without the model's real forecasts changing at all.

Watching the origin roll forward

The widget below does not use Lumen's sales. It ships with its own small built-in dataset and steps through 5 ordinary folds. Use it just for the stepping motion, not the random split underneath it: each time you click to the next fold, picture Lumen's origin jumping forward by 4 weeks instead, from week 52 on the first step to week 100 on the fifth, with the training window growing a little longer every time.

That growing-window feeling, a little more training data at every step while the forecast window keeps moving forward in time, is exactly what the loop you just ran in R was doing 13 times over.

Quick check: origins, horizons and the honest RMSE

What counts as leakage in a backtest

Leakage is any information reaching the model, directly or indirectly, that would not actually be available at the moment it has to produce that forecast. The shuffled fold from earlier was one form of it. But leakage keeps finding new ways into a backtest that looks, on the surface, like it respects the origin perfectly.

Four forms show up constantly in practice, and each one changes the reported accuracy number without changing what the model would actually do in production:

• A scaler, benchmark, or average computed from the whole series, instead of only the data available at each origin.
• A feature whose value could only have been computed using a week not yet reached.
• A model, or a setting of a model, chosen by looking at the very data the final score gets reported on.
• A missing gap between where training stops and the window actually being scored, when the real decision needs a longer lead time than that.

Each of these gets planted, one at a time, straight into the backtest you just ran, to show exactly what it does to the 8.48.

Leak one: a scaler fitted on the whole series

Lumen's weekly sales keep climbing, so the training mean at each origin grows with it. Early on, the training weeks average around 64 units. By the last origin, that average has climbed to 87.4.

A common way to report backtest accuracy is as a relative error: the forecast error divided by some average sales level, so it reads as a percentage instead of a raw unit count. The honest version divides each origin's error by that origin's OWN training mean, the only number known at that point in time. The leaky version divides by the mean of the WHOLE series instead, 89.6 units, a number that only exists once every week, including ones still in the future at most origins, has already been observed.

The forecasts never changed between these two numbers. Not one. The only thing that moved was the number sitting in the denominator, and that alone was enough to turn a 9.2% average error into a 7.7% one. A future-informed denominator always divides by a bigger number later in the series than it should, which is exactly why the leaky version comes out lower.

Leak two: a feature built from the future

The model you have been backtesting uses a trailing 4-week average: at week i, the mean of weeks i-4 to i-1, always strictly in the past. Suppose someone swaps that for a centred average instead: weeks i-2 to i+1, two behind and one ahead. It looks almost the same on paper, just a different window. But that window now reaches one and two weeks past week i itself.

The one origin whose horizon reaches week 104 gets dropped here, since its centred feature would need a week past the end of the data entirely. Over the remaining 12 origins, RMSE drops from an honest 8.32 to 7.15.

That drop is not noise. A centred average sits right on top of the week it is estimating, so it tracks Lumen's rising trend far more closely than an average that only ever looks backward and is always a little behind it. The backtest rewards the centred version for information it would never actually have at forecast time: the two weeks just after the one it is trying to call. In production, at the real origin, weeks i+1 and beyond have not happened yet, so this feature could never be built the way the backtest built it. This is the most common leak in practice, precisely because the buggy line looks so close to the honest one.

Leak three: a model selected on the test window

Here is a different kind of leak, one that does not touch any feature's definition at all. Add 15 columns of plain random noise, numbers with no connection whatsoever to Lumen's sales, one at a time to the honest model, and keep whichever one scores best.

Column 2 happens to score 8.30, a touch better than the plain model's 8.48. There is nothing in that column but random numbers. It won purely because 15 candidates were tried and the lowest score was kept, on the very same 13 origins the final number gets reported on. Try enough unrelated candidates against noisy data and one of them will look good by accident.

Here is the honest version of the same search. Pick the winning column using only the first 6 origins, then report that column's score on the remaining 7 origins, the ones that never had any say in which column got chosen.

Column 2 wins again on just the first 6 origins, so it gets carried forward and scored on the last 7. Its score there is 8.32. Now compare that to the plain model, with no noise column at all, scored on that exact same set of 7 origins: also 8.32. Once the column is validated on data it never influenced, it adds nothing. The apparent improvement from 8.48 down to 8.30 only ever existed because the same 13 origins did double duty as both the search ground and the scoreboard.

Leak four: the gap that should be there and is not

One more leak, and this one lives in the gap between training and the window being scored, not in any single feature.

Lumen's real decision every 4 weeks is not "what will next week look like." It is "how many lamps should I reorder for the coming month," a single number: the total sales over the next 4 weeks. Call that the reorder target. At week t, it is the sum of sales over weeks t+1 to t+4.

Here is the trap. If you train a model up to origin o and ask it to predict the reorder target AT week o, that target already needs to know sales from weeks o+1 through o+4, the exact weeks still in the future relative to the origin. Training data built this way has quietly absorbed sales figures from past the origin, through the targets of the training rows themselves, not through any feature. The fix is a gap: stop training h weeks earlier than the origin, so even the last training row's own target never reaches past what is actually known.

Training right up to the origin with no gap reports an RMSE of 15.37 on the reorder quantity. Leaving a proper 4-week gap, so training stops at o - 4 instead of o, raises that to 19.73. The no-gap version looks like the better model, but it only ever got there by letting its training targets quietly absorb sales figures from the very weeks it is about to be judged on.

Guarding a backtest against leakage

Four leaks, four questions worth asking before trusting any backtest number.

1. Was every scaler, benchmark, or average computed using only the data available up to each origin, never the whole series?
2. Could any feature's value have needed a week that had not been reached yet, the way a centred average does?
3. Was the model, or any setting of it, chosen by looking at the same window the final score gets reported on?
4. Does the gap between where training stops and the window being scored match the real horizon, the way the reorder target needed?

Here is every leak from this lesson side by side with its honest counterpart.

Read the first pair as a percentage and the other three as RMSE, since they are not on the same scale. What they share is the shape: the leaky bar is always the shorter, better-looking one, and the honest bar next to it is the number Lumen could actually count on.

Quick check: matching four backtests to their leak

Your turn: fix the leak in this backtest

Below is a rolling-origin loop that looks almost right. It is supposed to train only on weeks up to each origin, but one line undoes that completely: train_rows is fixed at 1:104, every single time through the loop, so every origin trains on the entire series, including weeks it is about to be scored on.

# Fixed: train_rows now depends on the origin, so training never reaches past it
leak_errors <- c()

for (o in origins) {
  train_rows <- 1:o
  train_df <- data.frame(sales = sales[train_rows], week = week_index[train_rows], trail = trail_avg[train_rows])
  fit <- lm(sales ~ week + trail, data = train_df)

  test_df <- data.frame(week = week_index[(o + 1):(o + h)], trail = trail_avg[(o + 1):(o + h)])
  forecasts <- predict(fit, newdata = test_df)

  actual <- sales[(o + 1):(o + h)]
  leak_errors <- c(leak_errors, actual - forecasts)
}

round(sqrt(mean(leak_errors^2)), 2)
#> [1] 8.52

Edit the train_rows line so training only ever reaches the origin, then press Check.

Training on the whole series every time let the model fit a trend line through the very weeks it was about to be judged on, which is why the leaky version reported 8.13, a better-looking number than the honest 8.48 from several steps back. Restricting train_rows to 1:o costs a little accuracy on paper. It is the only version that tells the truth about what this method would do with real, not-yet-happened weeks.

References

• Time series cross-validation - Hyndman, R.J. & Athanasopoulos, G., Forecasting: Principles and Practice (3rd ed.), section 5.10.
• On the use of cross-validation for time series predictor evaluation - Bergmeir, C. & Benitez, J.M. (2012), Information Sciences 191, 192-213.
• Advances in Financial Machine Learning - Lopez de Prado, M. (2018), the chapter on cross-validation and the purge/embargo gap.
• Leakage in data mining: formulation, detection, and avoidance - Kaufman, S., Rosset, S. & Perlich, C. (2012), KDD 2012.
• stretch_tsibble() - the tsibble package's expanding-window tool for building rolling-origin splits.

What you can do now

You built a rolling-origin backtest from scratch: an origin that only ever trains on the past, a horizon that matches the real decision, a window that expands as the origin rolls forward, and one honest RMSE, 8.48, that every leak in this lesson tried to improve on without improving anything real.

Then you planted four leaks, one at a time, into that same backtest, and watched each one move a reported number while Lumen's actual forecasts never changed at all: a scaler that divides by a future mean, a feature that reaches past the week it forecasts, a model chosen on the same data it gets scored on, and a missing gap before a target that needed one.

Next time a backtest looks unusually good, you now know exactly which four questions to ask it before you believe it.