Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 8 of 8

A Tuned, Stacked Model End to End

Across this course you have built a cabinet of very different learners: the maximum-margin SVM, kernels that bend a boundary, a discriminant blend, a Gaussian process that admits its own uncertainty, and behind you the whole random forest. This final lesson stops adding new machines and does the thing a working data scientist actually gets paid for: assemble them into one pipeline that tunes itself honestly, blends its members, and reports a number you can promise out loud.

Meet Marcus. He runs a used-car lot, and every trade-in is a negotiation that starts with one question: what is this car worth? He has a ledger of 900 past sales, each with the car's age, odometer reading, brand tier and accident history, and the price it eventually sold for. He wants a single model that prices a trade-in to within about a thousand dollars, and he wants to know that the "within a thousand dollars" is real and not a number he fooled himself into believing.

The diagram is the whole lesson in four moves. By the end you will be able to:

  • Explain the winner's curse: why the best score you find by trying many settings flatters you, and why trying even more settings makes it worse
  • Tune three model families with one honest recipe, every choice made from the training cars alone
  • Build the tuned stack end to end: out-of-fold columns, convex blend weights, a two-stage prediction
  • Evaluate the pipeline honestly, on one sealed test set touched exactly once, and say which number you are allowed to promise

Prerequisites: the stacking lesson (out-of-fold predictions and a meta-learner blend, which this lesson tunes and hardens), the kernel SVM (cost and gamma), Gaussian processes (the lengthscale), the random forest course (mtry and out-of-bag error), and k-fold cross-validation.

The one rule that governs everything

Seal the exam before you start

Everything that goes wrong in a modeling project goes wrong because some number that was supposed to estimate the future got to peek at the future first. So before we fit a single model, we make one commitment and never break it: a chunk of Marcus's cars is sealed away as a test set, an exam the pipeline does not get to see. Every later choice, which hyperparameters, which blend weights, is a function of the training cars only. The test set is opened once, at the very end, to grade the finished pipeline.

The score we will grade with is root mean squared error, the typical dollar miss:

\[ \text{RMSE} = \sqrt{\tfrac{1}{n}\sum_{i=1}^{n}\left(y_i - \hat{y}_i\right)^2} \]

where \(y_i\) is what car \(i\) actually sold for, \(\hat{y}_i\) is what the model guessed, and \(n\) is the number of cars scored. It is in the same units as price (thousands of dollars here), so an RMSE of 1.2 means "typically off by about \$1,200."

First, Marcus's ledger. Each lesson runs in a fresh R session, so we build the 900-car lot right here, with real structure baked in: cars depreciate at a rate that depends on brand tier, hard driving and accidents knock money off, and a couple of sharp effects (a nearly-new bump, an old-and-thrashed cliff) that only a flexible model can catch.

RInteractive R
library(e1071) # support vector machine library(kernlab) # Gaussian process library(randomForest) # random forest set.seed(42) n <- 900 age <- round(runif(n, 0.5, 12), 1) # years on the road km <- round(age * runif(n, 6, 14) + runif(n, 0, 15)) # thousand km on the odometer tier <- sample(c("budget", "mid", "premium"), n, TRUE) # brand tier accidents <- rbinom(n, 2, 0.15) # prior accidents: 0, 1 or 2 rate <- c(budget = 0.20, mid = 0.15, premium = 0.11)[tier] # how fast the tier loses value mult <- c(budget = 0.75, mid = 1.00, premium = 1.35)[tier] # its price level when new price <- (3 + 26 * exp(-rate * age) * mult - 0.015 * km) * 0.85^accidents + ifelse(age > 8 & km > 100, -3, 0) + # old-and-thrashed cliff ifelse(age < 1.5, 3, 0) + # nearly-new bump rnorm(n, 0, 1) # everything we did not measure lot <- data.frame(age, km, tier = factor(tier), accidents, price = round(pmax(price, 0.3), 2)) # price in $1,000s head(lot, 4) #> age km tier accidents price #> 1 11.0 87 premium 0 10.54 #> 2 11.3 76 budget 1 2.40 #> 3 3.8 41 premium 0 25.63 #> 4 10.1 76 budget 1 4.05

  

  





Now the split, and the promise. Two thirds of the lot, 600 cars, become the training ground; the other 300 are the sealed exam. We also define rmse() once, since every step from here scores with it.




  

    
RInteractive R

    
set.seed(101)
test_id <- sample(n, 300)          # 300 cars sealed away as the exam
train   <- lot[-test_id, ]         # 600 cars to build everything on
test    <- lot[test_id, ]

rmse <- function(actual, predicted) sqrt(mean((actual - predicted)^2))
c(train = nrow(train), test = nrow(test), sd_price = round(sd(train$price), 2))
#>    train     test sd_price
#>   600.00   300.00     8.36

    

      
      
    

    

  

  





The last number sets the bar. Prices in the training set have a standard deviation of \$8,360, so a model that just guessed the average price every time would miss by about that much. Anything we build has to beat 8.36 by a wide margin to be worth Marcus's trust. The widget shows the split, and the one way it all falls apart: flip the switch and watch a leaked feature turn the test score into a fantasy.