Train and test splits for temporal data

Today let's understand train and test splits for temporal data, using simple and practical examples.

Here is the running example for the whole lesson: quarterly beer production in Australia, from 1992 Q1 to 2010 Q2, in megalitres (a megalitre is a million litres). That's 74 quarters, published by the Australian Bureau of Statistics.

Look at the line. To check how good a forecasting model actually is, you need to hold some quarters back and compare the model's forecast against them, the same way you would hold back rows to test any other model. So here's the question this lesson answers: can you just grab any random 20 to 30% of these 74 quarters and call it the test set, the way you would with a table of unrelated rows? Or does the fact that this data moves through time change the rule?

What an ordinary train/test split assumes about your rows

Before getting to the beer series, it helps to see the ordinary rule first, the one you would use on a table of unrelated rows like customers or transactions, where the order of the rows means nothing at all.

A plain train/test split usually sets aside a random slice of the rows for testing and trains the model on the rest. The reason a random subset is fine there comes down to one specific assumption: every row is exchangeable with every other row. Shuffle the order of a customer table and nothing about it changes, so any random 30% you pick is just as representative of the whole table as any other random 30% would be.

The widget below is not the beer series. It's 200 made-up, unrelated rows, split into 60% train, 20% validation and 20% test, purely by row position, picked at random.

The coloured strip splits those 200 rows by role. Underneath it, the runnable R code shows the same idea in its simplest two-way form: idx <- sample(n, 0.7 * n) picks 70% of the row positions at random for training, and the rest becomes test. Flip the leak checkbox and watch what happens to the test accuracy: it jumps from an honest 0.78 to a suspicious 0.99, because a column that was secretly built from the answer snuck into training. That is a different kind of mistake from the one this lesson is about, but it shares the same root: touching or including information you would not really have at prediction time is what breaks a test set's honesty, whether that information leaks in through a planted column or through the order of the dates.

So remember, none of this used the beer series. But the same sample() idea is exactly what you would reach for by habit on the beer series too: pick 70% of the 74 quarters at random for training, and use the rest to test. The next step does precisely that, and shows you why it goes wrong.

What happens when that assumption meets an ordered series

Apply that exact same idea, sample() on row positions, to the beer series, and watch what happens.

First, build the beer series properly, as a tsibble: a data frame built for time series, where one column is explicitly marked as the time index.

Now split it the exact same way the widget did a moment ago: sample 70% of the 74 row positions at random for training, and put the rest in test.

Look closely at those two lines. 2009 Q4, the very last quarter of the whole series, landed in the random training set. And 1992 Q3, one of the very first quarters, landed in the random test set.

Think about what that means for a real forecasting job. A model trained partly on 2009 Q4 would then have its forecast compared against 1992 Q3, seventeen years earlier. That could never happen in a real deployment. In 1992, the year 2009 has not happened yet, so there is no way a model built and used in 1992 could ever have seen 2009's numbers.

This is called future leakage: information from the future sneaking into training, through nothing more sinister than an ordinary random split. Exchangeability, the assumption that made a random split safe in the last step, is exactly the assumption a time-ordered series like this one breaks. Order matters here, and a random split throws that order away.

The fix: the last-h holdout, and how to choose h

The fix is simple once you see the problem: cut by time, not by row position. Everything up to some cutoff date becomes the training set, and everything from that date onward becomes the test set, kept in the same order it always was in. This is called a last-h holdout, where h is the number of periods held out at the end for testing.

So how many quarters should h actually be? The rule is: h should match the forecast horizon you actually plan to use the model for. If you plan to forecast 8 quarters ahead in practice, test the model on its ability to forecast 8 quarters ahead, not 1 quarter and not 20.

There's a second condition too. Whatever is left for training, after you remove h quarters from the end, needs to be enough to estimate any recurring pattern in the series. Beer production repeats every 4 quarters, one full year, so training needs several of those 4-quarter cycles, at least 2 to 3 of them, to have a real chance at learning the pattern.

For the beer series, a forecast horizon of h = 8 (2 years) makes for a sensible holdout. That leaves 74 minus 8, or 66 quarters, 16.5 years, for training, far more than the 2 to 3 seasonal cycles a bare minimum would need.

Cutting the holdout with filter_index()

The tsibble package has a function built exactly for this: filter_index(). It keeps rows by date range, read straight off the tsibble's own time index, instead of by row number.

Its range is written as start ~ end, and a dot (.) on either side means "everything" in that direction. So . ~ "2008 Q2" reads as "from the very start through 2008 Q2", and "2008 Q3" ~ . reads as "from 2008 Q3 through the very end".

Cut the beer series into train and test using the 2008 Q2 boundary decided in the last step.

66 quarters in train, 8 in test, and 66 plus 8 is 74: every quarter in the series is accounted for exactly once, with no row dropped and none duplicated. beer_train runs from the series' very first quarter, 1992 Q1, straight through to 2008 Q2 without a single gap. beer_test picks up exactly where training left off, 2008 Q3, and runs to the series' last quarter, 2010 Q2.

Seeing the cut on the series itself

Colour that same 74-quarter series by the role each quarter now plays: the first 66 quarters, 1992 Q1 through 2008 Q2, as train, and the last 8, 2008 Q3 through 2010 Q2, as test.

Notice there is exactly one cut point on the whole chart, sitting right at the 2008 Q2 to 2008 Q3 boundary. Every train-coloured quarter sits to the left of it, and every test-coloured quarter sits to the right. Compare that with the random split from a few steps back, which scattered train and test quarters across the entire 74 quarters with no pattern at all.

A last-h holdout never does that. It cuts the series exactly once, at exactly one date, and everything before that date is train while everything from it onward is test.

Quick check: reading the split on the chart

Rolling origins: a preview of testing more than once

A single last-h holdout tells you how the model would have done on one particular 8-quarter stretch. But relying on just one test window means your accuracy estimate depends heavily on which 8 quarters you happened to pick. A rolling-origin evaluation fixes that by testing at several different cutoff dates instead of one, and averaging the results for a steadier estimate.

Here's a preview of the idea, using three origins, each one 4 quarters later than the last.

Origin 1 trains on 58 quarters, 1992 Q1 through 2006 Q2, and its test window is the 4 quarters right after that cutoff, 2006 Q3 through 2007 Q2. Origin 2 trains on 62 quarters, through 2007 Q2, and tests on 2007 Q3 through 2008 Q2. Origin 3 trains on 66 quarters, through 2008 Q2, the same cutoff from a few steps back, and tests on 2008 Q3 through 2009 Q2.

Each origin's test window always sits later in time than its own training window, the same rule the single holdout followed. Score the model at all three origins and average the three test errors, and you get a steadier estimate of how well the model forecasts than any single holdout could give you on its own.

There's more to rolling-origin evaluation than this preview shows: exactly how many origins to use, how to space them, and running the whole thing with a package built for it. That level of detail is out of scope here. The pattern itself, several origins instead of one, each with its test window later than its own training window, is the part worth taking away.

Test-set discipline: touch it once, at the end

There's one more way to break the exact same guarantee a random split broke, and it has nothing to do with how you cut the data.

The test window stands in for data you do not have yet, quarters that, in a real deployment, have not happened. That is only true as long as the test window's true values play no part in choosing or tuning the model. The moment you check test performance, adjust something about the model, and check test performance again, the test window has stopped standing in for the future. You have started fitting to it too, the same way training does, just more slowly and one peek at a time.

So the discipline is simple to state: fit the model and make every choice about it, which method, which settings, using only the training window. Then touch the test window's true values exactly once, at the end, to report how the model actually did. If you need to compare several models or settings along the way, use rolling-origin windows or a validation window carved out of training for that, and save the real test window for the final, one-time check.

Quick check: which split is safe for temporal data

Your turn: carve a 12-quarter holdout

beer, the 74-quarter tsibble, is still sitting in your session. Cut a different last-h holdout from it this time. Instead of the 8-quarter (h = 8) cut from a few steps back, use a 3-year horizon, h = 12, cut at 2007 Q2.

Build beer_train2 as everything through 2007 Q2, and beer_test2 as everything from 2007 Q3 onward, the same way beer_train and beer_test were built earlier.

# Carve a 12-quarter holdout: train through 2007 Q2, test from 2007 Q3 onward
beer_train2 <- beer |> filter_index(. ~ "2007 Q2")
beer_test2  <- beer |> filter_index("2007 Q3" ~ .)

nrow(beer_train2)
#> [1] 62
nrow(beer_test2)
#> [1] 12

References

• Forecasting: Principles and Practice (3rd ed.) - Hyndman and Athanasopoulos, the free online textbook. Section 5.8 covers training and test sets and evaluating point forecast accuracy; section 5.10 covers time series cross-validation.
• tsibble package reference - documentation for filter_index() and the rest of the tsibble grammar.
• Australian Bureau of Statistics, catalogue 8301.0.55.001, table 1 - the source of the beer production series used throughout this lesson.
• Bergmeir, C. and Benitez, J.M., "On the use of cross-validation for time series predictor evaluation," Information Sciences (2012) - why ordinary k-fold cross-validation misleads on time-ordered data.
• fabletools package reference - documentation for accuracy(), used to score a model on a held-out test window.

Quick recap

• A random train/test split assumes every row is exchangeable with every other row. That's fine for unrelated rows, but the beer series broke it: the same sample() split put 2009 Q4 in train and 1992 Q3 in test, seventeen years apart, a case of future leakage.
• The fix is a last-h holdout: cut by time, not by row position. h should match the forecast horizon you actually plan to use, and training should still retain several full seasonal cycles; for beer, h = 8 left 66 quarters, far more than enough of its 4-quarter cycle.
• filter_index() carves that holdout straight out of a tsibble by date range: beer_train through 2008 Q2, beer_test from 2008 Q3 onward, 66 and 8 quarters, no row dropped or duplicated.
• A rolling-origin evaluation previews testing at several cutoffs instead of one, each origin's test window later in time than its own training window, and averaging the results for a steadier estimate.
• The same guarantee a random split breaks can be broken again by peeking at the test window's true values more than once. Fit and choose using only the training window, and touch the test window exactly once, at the end.