Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 5 of 8

Stacking and the Super Learner

In Lesson 4 you gave a regression model a conscience: a Gaussian process that predicts a value AND admits how sure it is. This lesson does something different with your growing toolbox. Instead of crowning one winner and discarding the rest, it keeps several models and teaches a second model to blend them.

Meet Nadia. She runs a coffee cart outside a train station, and she has logged cups sold in 160 fifteen-minute windows: the hour of day, and the cups she sold. Demand has a morning rush, an evening rush, and a sharp jump at noon when the office across the road breaks for lunch. Nadia has built three models to forecast demand for her ordering app: a straight line, a smooth curve, and a regression tree. Her app has room for one. Which does she ship?

The surprising answer this lesson defends: none of them alone. By the end you will be able to:

  • Explain why picking a single best model leaves accuracy on the table, and why a plain average can even hurt
  • Build a stacking ensemble in R: out-of-fold predictions, a meta-learner, and the blend weights it learns
  • See why the meta-learner must be trained on out-of-fold predictions, not in-sample ones, and what goes wrong when it is not
  • Turn a stack into a Super Learner with convex weights, and score the whole thing honestly with nested cross-validation

Prerequisites: several strong, different learners already in hand (decision trees, SVMs, Gaussian processes from this course), k-fold cross-validation as an honest accuracy estimate, linear regression with lm(), and the overfitting idea from the ML Workflow course.

The interactive below is the lesson's payoff in miniature: three base learners and the stack built on top of them. Toggle to Test error and the stacked bar sits lowest; toggle to Blend weights to see how much the meta-learner leans on each. We will build every piece of this from scratch.

The problem

Three good models, one slot

Each lesson runs in a fresh R session, so we start by typing Nadia's log into the machine. We build her 160 windows and, separately, a larger fresh fortnight of data to score models on honestly, so no model is ever judged on rows it trained on.

RInteractive R
set.seed(1) n <- 160 cart <- data.frame(hour = round(runif(n, 6, 20), 2)) # 6am to 8pm mean_cups <- 12 + 18 * exp(-((cart$hour - 8)^2) / (2 * 1.2^2)) + # morning rush 14 * exp(-((cart$hour - 18)^2) / (2 * 1.5^2)) + # evening rush 8 * (cart$hour >= 12) # lunch office crowd: a hard jump cart$cups <- round(mean_cups + rnorm(n, 0, 4)) # cups sold in a 15-min window set.seed(2) # a fresh fortnight, held out for scoring m <- 300 test <- data.frame(hour = round(runif(m, 6, 20), 2)) test$cups <- round(12 + 18 * exp(-((test$hour - 8)^2) / (2 * 1.2^2)) + 14 * exp(-((test$hour - 18)^2) / (2 * 1.5^2)) + 8 * (test$hour >= 12) + rnorm(m, 0, 4)) rmse <- function(pred, truth) sqrt(mean((truth - pred)^2)) # our scoreboard: lower is better head(cart) #> hour cups #> 1 9.72 16 #> 2 11.21 12 #> 3 14.02 25 #> 4 18.71 26 #> 5 8.82 29 #> 6 18.58 34

  

  





Now Nadia's three models. They are deliberately simple and very different in shape: a line that can only tilt, a degree-5 polynomial that can bend smoothly, and a tree that predicts a flat value inside step-like ranges of the hour. These three stand in for the richer learners you built earlier (the SVMs, GPs and forests); stacking works exactly the same whatever the learners are, and these three run live and give the same numbers every time.




  

    
RInteractive R

    
library(rpart)
fit_lin  <- lm(cups ~ hour, data = cart)                      # a straight line
fit_poly <- lm(cups ~ poly(hour, 5), data = cart)             # a smooth curve
fit_tree <- rpart(cups ~ hour, data = cart,                   # a step function
                  control = rpart.control(cp = 0.002, minbucket = 5))

base_rmse <- c(
  linear = rmse(predict(fit_lin,  test), test$cups),
  poly   = rmse(predict(fit_poly, test), test$cups),
  tree   = rmse(predict(fit_tree, test), test$cups))
round(base_rmse, 3)
#> linear   poly   tree
#>  7.162  5.050  4.867

    

      
      
    

    

  

  





The tree wins, just, at 4.867 cups of typical error; the smooth curve is close behind; the straight line is hopeless (it cannot represent two rush hours with one slope). The naive move is to ship the tree and bin the other two. But look at WHERE each model is right before you throw anything away.




  

    
RInteractive R

    
library(ggplot2)
grid <- data.frame(hour = seq(6, 20, by = 0.1))
fits <- rbind(
  data.frame(hour = grid$hour, cups = predict(fit_lin,  grid), model = "linear"),
  data.frame(hour = grid$hour, cups = predict(fit_poly, grid), model = "poly (smooth)"),
  data.frame(hour = grid$hour, cups = predict(fit_tree, grid), model = "tree"))
ggplot(cart, aes(hour, cups)) +
  geom_point(colour = "grey70", size = 1.2) +
  geom_line(data = fits, aes(hour, cups, colour = model), linewidth = 1) +
  labs(title = "Three models, each right in a different place",
       x = "hour of day", y = "cups sold (15-min window)") +
  theme_minimal(base_size = 13)

    

      
      
    

    

  

  





Press Run. The smooth curve traces the two rushes gracefully but rounds off the sharp noon jump. The tree nails the noon jump (a clean step) but climbs the rushes in coarse stairs. The line just splits the difference everywhere. Each model is wrong exactly where another is right. Throwing two of them away throws away the corrections they could make. Keeping all three is the whole idea.