Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 8 of 11

Uplift and Heterogeneous Effects

In Lesson 7 you measured the effect of a policy on a single treated city, one honest average number. Look back at every method in this course and they all share that goal: matching, weighting, difference-in-differences, instruments, synthetic control, each one hands you one number, the average effect across everybody.

But an average can lie by telling the truth. A retention email that lifts renewal by 15 points on average can still be helping some customers a lot, doing nothing for others, and actively pushing a third group out the door. Send it to everyone and you pay to annoy the very people it hurts. This lesson is about opening up that average: predicting the effect for each customer separately, so you treat the people a policy actually helps and leave the rest alone.

By the end of this lesson you will be able to:

  • Explain why one average effect hides who is helped and who is hurt, and name the four customer types
  • Build a T-learner in R, one model per treatment arm, to score each customer's individual uplift
  • Validate an uplift model even though you can never see any single customer's true effect
  • Read a Qini curve to decide how many customers to target, and know why the answer is often not "everyone"

Prerequisites: Lesson 1 (potential outcomes \(Y(1)\) and \(Y(0)\)) and Lesson 3 (why a naive comparison is biased). You can read a glm(y ~ x, binomial) fit as "give me the probability," and you know base R subsetting.

The setup

One email, one average, 4,000 subscribers

Meet Nadia, who runs retention at Cadence, a music-streaming service. Every month a batch of subscribers reach their renewal date, and some are wavering. Nadia can send them a "20% off your next three months, stay with us" email. She wants one thing: to send it to the customers it helps, and not waste it (or worse) on the rest.

So she ran a clean experiment. She took 4,000 at-risk subscribers and flipped a coin for each one: a random half got the email, the other half got nothing. Then she waited to see who renewed. Because assignment was random, the two halves are comparable, exactly the randomized setup Lesson 1 fought to recover from messy observational data.

Each lesson runs in a fresh R session, so we build Nadia's experiment ourselves. The one feature we track is engagement, a standardized score of how much the subscriber used Cadence lately: 0 is a typical user, +1 a heavy listener, -1 someone who has barely opened the app. Because the data is simulated, we also know the hidden truth we will spend the lesson recovering: the email's true effect varies by customer.

RInteractive R
set.seed(5) n <- 4000 engagement <- runif(n, -1, 1) # standardized recent-usage score, -1 (cold) .. +1 (hot) email <- rbinom(n, 1, 0.5) # randomized: 1 = got the retention email, 0 = did not tau <- 0.35 * engagement + 0.15 # the HIDDEN true per-customer uplift (we know it; Nadia does not) renewed <- rbinom(n, 1, pmin(pmax(plogis(-0.2 + 0.5 * engagement) + email * tau, 0), 1)) head(data.frame(engagement = round(engagement, 2), email, renewed)) #> engagement email renewed #> 1 -0.60 1 0 #> 2 0.37 0 1 #> 3 0.83 0 1 #> 4 -0.43 1 1 #> 5 -0.79 1 1 #> 6 0.40 0 0

  

  





Now the number Nadia would actually report. Because the email was randomized, the honest average effect is just the renewal rate of the emailed group minus the renewal rate of the not-emailed group.




  

    
RInteractive R

    
round(c(treated = mean(renewed[email == 1]),
        control = mean(renewed[email == 0])), 3)
#> treated control
#>   0.600   0.452
round(mean(renewed[email == 1]) - mean(renewed[email == 0]), 2)   # the average treatment effect
#> [1] 0.15

    

      
      
    

    

  

  





The email lifts renewal from 45% to 60%, a +15 point average effect. A clear win. So send it to everyone?




  

    
RInteractive R

    
library(ggplot2)
rates <- data.frame(group = c("No email (control)", "Retention email"),
                    rate  = c(mean(renewed[email == 0]), mean(renewed[email == 1])))
ggplot(rates, aes(group, rate, fill = group)) +
  geom_col(width = 0.6, show.legend = FALSE) +
  geom_text(aes(label = paste0(round(rate * 100), "%")), vjust = -0.4) +
  scale_fill_manual(values = c("grey72", "#1c2c4f")) +
  labs(x = NULL, y = "renewed", title = "The email lifts renewal by about 15 points on average")