Navigate
Courses Tutorials Practice Pricing Tools Upgrade

Estimating an event's effect with a counterfactual forecast

Today let's understand how to work out what an event actually did to a number you track every day, using a real example you can run yourself.

Daybreak Coffee is an online coffee-subscription retailer. For 90 days it tracked its daily online orders, and on day 61 it launched a podcast advertising campaign. The 60 days before the campaign averaged 541.9 orders a day. The 30 days after averaged 668.2 orders a day.

Here is the whole 90-day series, colored by whether the day fell before or after the campaign.

Orders step up noticeably right where the color changes at day 61. But a weekly pattern was already pushing orders up and down before the campaign ever started, and the series was already drifting upward day by day on top of that. So the question this lesson answers is: how much of that step up is the campaign, and how much would have happened anyway?

What a counterfactual answers that a before and after average cannot

The gap between those two averages is 668.2 minus 541.9, which comes to 126.3 orders a day. It is tempting to call that the campaign's effect. But it is not, because both periods already carried patterns that had nothing to do with the campaign.

Build the series the way it was actually generated, and two of those patterns show up right in the code.

RInteractive R
# Build Daybreak Coffee's 90 days of orders and compare the before/after averages set.seed(110) day <- 1:90 day_of_week <- ((day - 1) %% 7) + 1 # 1 = Monday ... 7 = Sunday weekly_add <- c(0, 15, 10, 5, 20, 55, 60)[day_of_week] # Friday and Saturday run busiest baseline <- 500 + 0.6 * day + weekly_add noise <- rnorm(90, mean = 0, sd = 18) orders_obs <- baseline + noise post_days <- 61:90 ramp <- pmin((post_days - 60) / 7, 1) # the lift ramps in over the first week orders_obs[post_days] <- orders_obs[post_days] + 0.20 * baseline[post_days] * ramp orders_obs <- round(orders_obs) round(c(pre_mean = mean(orders_obs[1:60]), post_mean = mean(orders_obs[61:90]), naive_gap = mean(orders_obs[61:90]) - mean(orders_obs[1:60])), 1) #> pre_mean post_mean naive_gap #> 541.9 668.2 126.3 

  

  





weekly_add adds the Friday and Saturday peak to every single day, before the campaign and after it alike. The 0.6 * day term is a small steady climb that was already built into the series from day one. So part of that 126.3 naive gap is just the weekly pattern and the drift landing differently across the two periods, nothing to do with the campaign.



To separate the two properly, name the pieces.



    


  • The response series is the number being measured over time. Here it is orders_obs, the daily order count.
    • 


  • The intervention date is the day the event happened: day 61.
    • 


  • The pre-period is every day before the intervention: days 1 to 60.
    • 


  • The post-period is every day from the intervention onward: days 61 to 90.
    • 


  • The counterfactual is what the response series would have looked like in the post-period if the intervention had never happened.
    • 





The campaign's effect is not post-period average minus pre-period average. It is actual minus counterfactual, day by day, and the counterfactual has to come from a model that never saw the campaign.