Navigate
Courses Tutorials Practice Pricing Tools Upgrade

Time-series cross-validation with rolling origins

Today let's look at how to judge a forecasting method fairly, using 5 years of monthly orders from a small online bookstore.

Nordic Books ships about 200 orders a month, and that number climbs every October through December as shoppers buy gift copies for the holidays. Below is all 5 years of it, January 2021 through December 2025, 60 months in total.

The first 48 months sit in one color, and the last 12 sit in a second color. That block of 12 held-out months is the one most people reach for first: fit a model on everything before it, then check how close the model's guesses land against those real 12 months.

One split, one score

Let's do exactly that: fit a model on the first 48 months and score it against the last 12.

The model here is SNAIVE, short for seasonal naive. It forecasts each month by repeating whatever Nordic Books sold in that same month one year earlier. December gets last December's number, January gets last January's, and so on. It is the standard first model to try on a series with a clear yearly pattern like this one.

Press Run.

RInteractive R
# Build five years of Nordic Books' orders, then score one train/test split library(tsibble) library(fable) library(fabletools) library(dplyr) set.seed(42) n <- 60 trend <- seq(180, 260, length.out = n) month_num <- rep(1:12, length.out = n) bump <- ifelse(month_num == 10, 10, ifelse(month_num == 11, 35, ifelse(month_num == 12, 55, 0))) orders <- round(trend + bump + rnorm(n, 0, 8)) nb <- tsibble( month = yearmonth("2021 Jan") + 0:(n - 1), orders = orders, index = month ) train1 <- nb %>% filter(month <= yearmonth("2024 Dec")) fc1 <- train1 %>% model(snaive = SNAIVE(orders)) %>% forecast(h = 12) accuracy(fc1, nb) %>% select(RMSE, MAE, MAPE) #> # A tibble: 1 × 3 #> RMSE MAE MAPE #> <dbl> <dbl> <dbl> #> 1 22.5 19.8 7.70

  

  





train1 keeps only the first 48 months. SNAIVE(orders) fits on that alone, and forecast(h = 12) produces one guess for each of the 12 held-out months. accuracy() then compares those 12 guesses against the 12 real values and boils the whole comparison down to a handful of numbers.



RMSE (root mean squared error) came out at 22.5, and MAE (mean absolute error) at 19.8. Both are in orders, the same unit as the data itself, so on average SNAIVE's monthly guess missed by somewhere around 20 orders. MAPE puts that same miss on a percentage scale: 7.70%.



That is one number for one particular way of drawing the line between train and test. Change where that line falls by even a month or two, and SNAIVE would land on a different set of 12 test months, each with its own mix of easy and hard ones to predict, and the RMSE would move too.