Navigate
Courses Tutorials Practice Pricing Tools Upgrade

Boosted Trees for Forecasting

Today let's forecast monthly airline passengers with gradient boosted trees, and find out what a tree model can and cannot do with a series that keeps rising.

The data is the number of international airline passengers per month, in thousands, from Jan 1949 to Dec 1960. That is 144 months, and R ships it as the built-in AirPassengers series. It climbs from 112 in Jan 1949 to 622 in Jul 1960, and the gap between the low months and the high months of each year widens as the level rises.

Now suppose it is Dec 1958 and you have to plan demand. You fit a model on the 120 months up to Dec 1958, forecast the 24 months of 1959 and 1960, and then score the forecast against what really happened.

The line labelled train is the 120 months a model is fit on. The line labelled test is the 24 months kept out of every fit, and every forecast in this lesson is scored against it.

Turning a monthly series into a table of lags

A boosted tree model needs a table: one row per example, one column for the target, the value we want to predict, and the other columns for the features, the values we predict it from. A time series is a single column, so we build the features out of the series' own past. A feature made from an earlier value of the series is called a lag.

For each month we use five features:

  • lag_1, lag_2 and lag_3: the passengers 1, 2 and 3 months earlier.
  • lag_12: the passengers in the same month one year earlier.
  • month: the calendar month, from 1 for Jan to 12 for Dec.

The first three lags carry the recent level, lag_12 carries the yearly pattern, and month identifies which part of the year a row belongs to. The function below builds this table for any series, and the last line prints the first 16 months.

RInteractive R
# Build a table where each month's row holds the passengers from earlier months passengers <- as.numeric(AirPassengers) make_lag_table <- function(series) { lag_by <- function(k) c(rep(NA, k), head(series, -k)) data.frame( y = series, month = rep(1:12, length.out = length(series)), lag_1 = lag_by(1), lag_2 = lag_by(2), lag_3 = lag_by(3), lag_12 = lag_by(12) ) } lag_table <- make_lag_table(passengers) lag_table[1:16, ] #> y month lag_1 lag_2 lag_3 lag_12 #> 1 112 1 NA NA NA NA #> 2 118 2 112 NA NA NA #> 3 132 3 118 112 NA NA #> 4 129 4 132 118 112 NA #> 5 121 5 129 132 118 NA #> 6 135 6 121 129 132 NA #> 7 148 7 135 121 129 NA #> 8 148 8 148 135 121 NA #> 9 136 9 148 148 135 NA #> 10 119 10 136 148 148 NA #> 11 104 11 119 136 148 NA #> 12 118 12 104 119 136 NA #> 13 115 1 118 104 119 112 #> 14 126 2 115 118 104 118 #> 15 141 3 126 115 118 132 #> 16 135 4 141 126 115 129

  

  





lag_by(k) moves the series down by k rows and puts NA in the first k rows, because those months have no earlier value. That is why lag_12 is NA for the whole first year.



Row 13 is Jan 1950. It has 115 passengers, and its features are 118 for Dec 1949 (lag_1), 104 for Nov 1949 (lag_2), 119 for Oct 1949 (lag_3) and 112 for Jan 1949 (lag_12). Every feature in a row comes from an earlier month, so no row's features include its own target.



Rows 1 to 12 have no lag_12, so the usable rows are 13 to 144. We take rows 13 to 120, Jan 1950 to Dec 1958, as the training rows, and rows 121 to 144, Jan 1959 to Dec 1960, as the test rows.




  

    
RInteractive R

    
# Keep the months that have every lag and split them into training and test rows
train <- lag_table[13:120, ]
test  <- lag_table[121:144, ]

c(train_rows = nrow(train), test_rows = nrow(test))
#> train_rows  test_rows
#>        108         24

    

      
      
    

    

  

  





The 12 months lost to lag_12 leave 108 training rows. The 24 test rows are the months we are going to forecast.