Rolling-origin backtests without leakage

Today let's understand how to test a forecasting method properly, and the sneaky ways that test can make it look better than it is without you ever noticing.

Lumen & Co is a small online retailer, and its best-selling product is a desk lamp. Below are its weekly unit sales for the last 104 weeks, two full years.

Sales start around 40 units a week and climb to around 140, with the usual week-to-week bounce along the way. Over the first year that averages out to about 64 units a week. By the second year it has risen to about 115.

Suppose you build a forecasting method for this series, and you want to know how accurate it really is before you trust it with real reorder decisions. That rising line above is the only data this lesson uses. Everything from here on asks one question about it: can you trust a backtest run on it, or can the backtest quietly overstate how good that method really is?

Why an ordinary shuffled fold is itself a leak on a time series

If you have tested a model before, you have probably used cross-validation: split the rows into a handful of folds, hold one fold out as a test set, train on the rest, and repeat until every fold has had a turn as the test set. The rows land in their folds at random, so each fold is just some random slice of the whole dataset.

That works fine when the rows do not have an order to them. But Lumen's 104 weeks are not interchangeable rows. Week 30 comes before week 50, and a forecast for week 30 is only honest if it was built without ever touching week 50.

Here is what a random fold does to that order. Suppose you split Lumen's 104 weeks into a random 80% training set and a 20% test set, and week 30 happens to land in the test set, the week you are trying to forecast.

week 30            -> test
weeks 31 through 40 -> all land in the training set

Ten weeks that come right after week 30 on the calendar, weeks that have not even happened yet relative to week 30, end up training the very model being asked to forecast it. The model gets to peek at the future before it makes its guess, and that is the simplest leak there is.

So the fix is not a better random split. It is to stop assigning weeks to folds at random altogether, and build the split around the calendar instead.