Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 6 of 7

Comparing Models Statistically

You have a fair yardstick now. In Lesson 5 you learned to score a forecast with a metric that matches the decision. So you train two models to forecast Priya's daily cake sales, you measure them, and one comes out ahead. Ship it, right?

Not yet. That single number came from one test split, one particular draw of which rows you happened to hold out. Change the draw and the winner can change. This lesson teaches you to separate a real difference between two models from the ordinary luck of the split, so that when you say "Model B is better" you can mean it.

By the end you will be able to:

  • Explain why a single test-split gap is partly luck, not a verdict
  • Turn cross-validation into paired per-fold differences for two models
  • Test whether the average difference is real, and read the p-value honestly

Prerequisites: you can run R and index a data frame, you know RMSE from Lesson 5, and what a cross-validation fold is from Lesson 1.

The trap

One test split is just one draw

Back at Priya's bakery. She wants to automate her daily sales forecast, and she is choosing between two models. Model A predicts the day's cake sales from the temperature alone. Model B adds two columns Priya knows matter: whether it is a weekend, and whether a promotion is running. Both are honest models; the only question is which one to deploy.

Each lesson runs in a fresh R session, so let us build four months of bakery days right here, then measure both models the usual way: train on most of the days, test on a held-out slice, and compare their RMSE (the average size of a forecast miss, from Lesson 5).

RInteractive R
set.seed(42) n <- 96 bakery <- data.frame( temperature = round(runif(n, 5, 30), 1), # daily high, in Celsius weekend = rbinom(n, 1, 2/7), # 1 on a weekend day promo = rbinom(n, 1, 0.25), # 1 if a promotion ran humidity = round(runif(n, 40, 90)) # unrelated to sales (used later) ) bakery$sales <- round(pmax( 40 + 1.8 * bakery$temperature + 24 * bakery$weekend + 17 * bakery$promo + rnorm(n, 0, 18), 0)) # sales = temp + weekend + promo + noise rmse <- function(actual, pred) sqrt(mean((actual - pred)^2)) # score both models on ONE random train/test split split_gap <- function(seed) { set.seed(seed) test <- sample(n, 24) # hold out 24 days as the test set tr <- bakery[-test, ]; te <- bakery[test, ] a <- rmse(te$sales, predict(lm(sales ~ temperature, data = tr), te)) b <- rmse(te$sales, predict(lm(sales ~ temperature + weekend + promo, data = tr), te)) round(c(rmse_A = a, rmse_B = b, gap = a - b), 2) } split_gap(4) # one particular split #> rmse_A rmse_B gap #> 19.90 16.06 3.84

  

  





On that split Model B wins by 3.84 cakes of RMSE. Convincing. But "that split" was one arbitrary choice of which 24 days to hold out. Watch what happens when we hold out a different 24 days:




  

    
RInteractive R

    
split_gap(3)   # a different split, same two models, same data
#> rmse_A rmse_B    gap
#>  19.40  19.28   0.12

    

      
      
    

    

  

  





Same two models, same 96 days, and now the gap has collapsed to 0.12, essentially a tie. Nothing changed except which rows landed in the test set. The 3.84 was never a fixed property of the models; a large chunk of it was luck. From one split you genuinely cannot tell whether B's true edge is worth deploying or close to nothing.