Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 9 of 11

Double Machine Learning

In Lesson 8 you used machine-learning models to predict a different treatment effect for every customer. Here is the sharp question that raises, made concrete. BrightLearn, an online course platform, sells an hour-by-hour tutoring add-on, and its team wants a single number: how many extra exam points does one weekly hour of tutoring actually cause? The obvious move is to throw a flexible model like a random forest at the data and read the effect off. Do that and the answer comes back badly wrong, more than double the truth, because the model quietly biases it. This lesson shows the fix, a way to keep the random forest for the messy parts and still recover one honest treatment effect, and it is one of the most useful ideas in modern causal inference.

By the end you will be able to:

  • Explain why plugging a machine-learning model straight into a causal estimate biases it
  • Use partialling-out to turn two prediction models into one honest treatment effect
  • Cross-fit two random forests in R to recover an effect that ordinary regression gets badly wrong

Prerequisites: Lessons 1 and 2 of this course (confounding and potential outcomes, the propensity score and covariate adjustment); you can fit lm and a random forest in R and read a coefficient with its confidence interval.

The running example

The number that lies

We met BrightLearn on the cover: an online course platform whose optional add-on is live one-on-one tutoring, booked by the hour. Its product team asks a question worth real money: how many extra exam points does one weekly hour of tutoring actually cause? Get it right and they know what the add-on is worth; get it wrong and they price it, or push it, on a fantasy.

Each lesson runs in a fresh R session, so we build BrightLearn's data right here. We plant the true answer ourselves, 2 points per weekly hour, so at the very end we can check who recovered it. Four measured student traits drive everything, and they act through sharp thresholds, not neat straight lines (real behaviour usually does).

RInteractive R
set.seed(2024) n <- 1500 ability <- runif(n) # aptitude score, 0 (low) to 1 (high) prior <- runif(n) # grade in the previous course, 0 to 1 study <- runif(n) # weekly self-study, 0 to 1 engage <- runif(n) # how much they use the platform, 0 to 1 # Weekly tutoring HOURS booked (the treatment). Keener, abler students book more. tutoring <- 4 + 4 * (ability > 0.5) + 3 * (prior > 0.6) + 2 * (study < 0.3) + rnorm(n) # Final exam SCORE. The TRUE causal effect is 2 points per weekly hour (we set it). score <- 2 * tutoring + 30 + 15 * (ability > 0.5) + 10 * (prior > 0.6) + 8 * (study < 0.3) + 6 * engage + rnorm(n, sd = 3) dat <- data.frame(score, tutoring, ability, prior, study, engage) round(head(dat, 4), 2) #> score tutoring ability prior study engage #> 1 66.06 8.00 0.84 0.41 0.99 0.74 #> 2 42.86 4.29 0.32 0.16 0.87 0.42 #> 3 81.18 11.53 0.68 0.74 0.43 0.95 #> 4 63.00 7.83 0.70 0.49 0.61 0.14

  

  





Now the obvious thing: regress score on tutoring and read the slope.




  

    
RInteractive R

    
round(coef(lm(score ~ tutoring, data = dat))[["tutoring"]], 2)
#> [1] 5.17

    

      
      
    

    

  

  





The data says each weekly hour is worth 5.17 points. But we planted the truth at 2. The naive number is off by more than a factor of two, and nothing in the regression output warns you.



Key Insight
This is not sampling noise, more data would keep landing near 5, not near 2. The trouble is that the students who book more tutoring are already different, and those same differences lift their scores. That is confounding, and defeating it is the whole point of this lesson.