Correlation, Causation and Potential Outcomes
Riverside Books, a small online bookshop, emailed a $10 coupon to some of its customers. The next month, the customers who got a coupon spent far more than the ones who did not. Marketing wants to declare victory: the coupon works, send it to everyone.
Before they do, look at the pattern below. Each dot is a customer: how loyal they were last year (across the bottom) against what they spent this month (up the side). It is a strong, real correlation, and it is exactly the kind of picture that fools people. This whole lesson is about the hard question hiding inside it: when does a pattern like this actually mean one thing causes another?
By the end of this lesson you will be able to:
- Explain why a correlation between two things does not prove that one causes the other
- Define a causal effect honestly, using the potential-outcomes idea at the heart of modern causal inference
- See, in real R, why the coupon's raw $18 "effect" is mostly an illusion, and how randomizing the coupon recovers its true $8 effect
Prerequisites: you can run R and take a mean, and you can read a scatterplot. Every new term is defined as it appears.
The gap Riverside Books sees
Let us put real numbers on the story. We will build Riverside Books' customer records right here, because each lesson runs in its own fresh R session. Each row is one customer, with three things we observe: their loyalty last year (a score centered at 0, where positive means more engaged), whether marketing emailed them a coupon, and their spend the next month in dollars.
Because this is a simulation, we can also secretly store two extra columns, y0 and y1: the spend we would see for that customer without and with a coupon. Hold on to those, they are the whole trick of this lesson, and in real life you never get to see them.
There it is, the pattern marketing noticed: customers who got a coupon spent about $58 on average, against $40 for those who did not, a gap of about $17.80. The two variables, "got a coupon" and "spend," clearly move together. In the language of the earlier EDA lessons they are correlated: knowing one tells you something about the other.
A correlation is just a measured association. For two numbers it is summarized by Pearson's \(r\),
\[ r = \frac{\sum_i (x_i - \bar x)(y_i - \bar y)}{\sqrt{\sum_i (x_i - \bar x)^2}\;\sqrt{\sum_i (y_i - \bar y)^2}} \]
where \(x_i\) and \(y_i\) are the two measurements for customer \(i\), and \(\bar x, \bar y\) are their averages. The \(r\) on the cover scatter is about 0.9. But \(r\), and the $17.80 gap, are both silent about one thing: why.
A confounder is hiding in the gap
Here is what the average hides. Look again at how the coupons were handed out: plogis(1.2 * loyalty) means the more loyal a customer already was, the more likely marketing was to email them one. The coupon did not fall from the sky at random. It landed on the customers who were already going to spend more anyway.
So the coupon group and the no-coupon group were not comparable to begin with. The coupon group was stacked with loyal big spenders. Some of their extra $17.80 is the coupon doing real work, and some of it is just who was in the group. That second part has a name.
A confounder is a third variable that influences both who gets the treatment and the outcome. Here it is loyalty: it pushed customers toward getting a coupon, and it pushed their spending up on its own. Whenever a confounder is in play, a plain correlation blends the true effect with the confounder's shadow.
In fact, any time two things A and B are correlated, one of a few stories must be true:
- A really does cause B (the coupon lifts spending).
- B causes A (the reverse: big spenders get offered more coupons).
- A confounder C causes both (loyalty drives coupons and spending).
- It is a coincidence in a small sample.
Spot the flaw
Riverside's team notices a second pattern: customers who installed the shop's app spend more than those who did not, and someone suggests the app itself drives the extra spending. What is the most important worry before believing the app causes higher spend?
Two worlds for one customer
To even ask whether the coupon caused Maya to spend more, we first have to say what "caused" means for one person. The clean way is to imagine two parallel worlds.
In one world, Maya gets the coupon; call what she spends there \(Y_{\text{Maya}}(1)\), read "Maya's outcome under treatment." In the other world everything about Maya is identical but she gets no coupon; call that \(Y_{\text{Maya}}(0)\), "her outcome under control." These two numbers are her potential outcomes, and her personal causal effect of the coupon is simply the difference:
\[ \tau_{\text{Maya}} = Y_{\text{Maya}}(1) - Y_{\text{Maya}}(0) \]
where \(\tau\) (tau) is the effect for that one customer: the extra dollars the coupon caused for her, with everything else held fixed. More generally, for customer \(i\) we write \(W_i = 1\) if they got the coupon and \(0\) if not, and their effect is \(\tau_i = Y_i(1) - Y_i(0)\).
Now the catch that makes causal inference hard. Picture three customers:
| Customer | Coupon? | Spend WITH, Y(1) | Spend WITHOUT, Y(0) | Effect |
|---|---|---|---|---|
| Maya | yes | 62 (seen) | ? never happened | ? |
| Leo | no | ? never happened | 41 (seen) | ? |
| Ada | yes | 70 (seen) | ? never happened | ? |
For every customer, exactly one column is real and the other is a counterfactual: a world that did not happen. Maya got the coupon, so we see her \(Y(1) = 62\) but never her \(Y(0)\). Leo got none, so we see his \(Y(0) = 41\) but never his \(Y(1)\). The effect column is all question marks, because an effect is a difference between two numbers and we only ever get one of them.
The effect we can chase, and the bias in the gap
If we can never get one person's effect, what can we get? The average. We give up on Maya's personal \(\tau\) and aim for the average treatment effect across all customers:
\[ \text{ATE} = \mathbb{E}\big[\,Y_i(1) - Y_i(0)\,\big] = \mathbb{E}[Y_i(1)] - \mathbb{E}[Y_i(0)] \]
where \(\mathbb{E}[\cdot]\) means "the average over all customers." In words: if everyone got a coupon versus if no one did, how much would average spend change? For Riverside that is the number worth knowing.
So why can we not just use the $17.80 gap for it? Write the gap in potential-outcome terms. What we actually computed was the average observed spend of the coupon group minus that of the no-coupon group. Because an observed spend is \(Y(1)\) for the treated and \(Y(0)\) for the untreated, that gap splits cleanly in two:
\[ \underbrace{\mathbb{E}[Y \mid W=1] - \mathbb{E}[Y \mid W=0]}_{\text{the naive gap}} = \underbrace{\mathbb{E}[Y(1) - Y(0) \mid W=1]}_{\text{real effect on the treated}} + \underbrace{\mathbb{E}[Y(0) \mid W=1] - \mathbb{E}[Y(0) \mid W=0]}_{\text{selection bias}} \]
The first piece is the honest effect for the customers who got a coupon. The second, selection bias, is the difference in the no-coupon spend between the two groups: how far apart they would have been even if no coupon had ever been sent. Since loyal big spenders were the ones handed coupons, that term is positive, and it inflates the gap. Because this is a simulation, we can measure all three numbers directly.
There is the whole illusion in three numbers. The coupon's true effect is $8. Selection bias adds $9.82. Together they make the $17.82 gap marketing saw (the same gap, the 40.2 and 58.0 you saw were just rounded), more than half of which is not the coupon at all, only loyal customers being loyal.
Measure the true effect
Because Riverside's records are simulated, both potential outcomes are sitting in books: y1 (spend with a coupon) and y0 (spend without). The true average treatment effect is just the average of their per-customer difference. Fill in the blank, then check it.
Show answer
ate <- mean(books$y1 - books$y0)
ate
#> [1] 8Randomization: the great equalizer
Selection bias came from letting loyalty decide who got a coupon. So take that decision away from loyalty, and away from everything else about the customer, and hand it to a coin.
If the coupon is assigned by a fair coin flip, then whether a customer is treated has nothing to do with their potential outcomes. In symbols, \(W \perp \big(Y(0), Y(1)\big)\), where \(\perp\) means "is independent of." That independence forces \(\mathbb{E}[Y(0) \mid W=1] = \mathbb{E}[Y(0) \mid W=0]\): the two groups have the same no-coupon spend on average, so the selection-bias term collapses to zero. What is left of the naive gap is exactly the ATE.
That is the entire logic of a randomized experiment (an A/B test), in four moves:
Let us prove it on the very same customers. We keep each person's planted y0 and y1, but this time we ignore loyalty and flip a coin for the coupon.
The simple difference in means is now $8.10, essentially the true $8. Same customers, same coupon, same code for the gap, and one changed thing: how the coupon was assigned. Randomizing made the groups comparable, and the bias vanished.
Is the effect beyond chance?
Randomizing gave us an estimate of $8.10. But the coin could have fallen a little unevenly, so we owe one more question: could a difference this big come from luck alone, if the coupon truly did nothing?
That is a hypothesis test. We suppose a sceptic's world in which the coupon has zero effect (the null hypothesis), and ask how surprising our $8.10 would be there. The p-value is that surprise: the probability of seeing a gap at least this large if the true effect were really zero. Drag the observed statistic below and watch its tail area, the p-value, shrink as the estimate moves further from zero.
For our experiment, R runs the test directly.
The p-value is under \(2.2 \times 10^{-16}\), astronomically small, so we reject "the coupon does nothing." (The t-value is negative only because R reported group 0 minus group 1; the coupon still raised spend, by about $8.) The 95% interval, roughly $7.1 to $9.1, comfortably brackets the true $8 and excludes zero.
Why the coin flip works
In Riverside's experiment, assigning the coupon by a coin flip let the plain difference in group means estimate the true causal effect, which the original targeted rollout could not. Why does randomizing fix it?
References
Four authoritative places to take this further:
- Hernan and Robins, Causal Inference: What If (free PDF) - the standard modern text; Chapters 1 and 2 define potential outcomes and the fundamental problem exactly as we did here.
- Holland (1986), Statistics and Causal Inference, JASA - the paper that named the "fundamental problem of causal inference" and formalized the potential-outcomes view.
- Cunningham, Causal Inference: The Mixtape (free, online) - the potential-outcomes chapter, worked slowly from intuition with code.
- Rubin (1974), Estimating causal effects of treatments (DOI) - the origin of the potential-outcomes framework these ideas rest on.
Lesson 1 complete
You now have the frame that all of causal inference is built on. A correlation, even a strong and highly significant one, blends any real effect with selection bias whenever a confounder decides who gets treated, so it can never prove cause by itself. A causal effect is defined honestly through potential outcomes, \(Y_i(1)\) and \(Y_i(0)\), one of which is always a missing counterfactual, which is why we chase the average treatment effect instead of any one person's. And randomizing the treatment makes the groups comparable, zeroing the selection-bias term so a plain difference in means estimates the effect, exactly as you watched the coupon's illusory $17.82 gap collapse back to its true $8.
Next, Lesson 2: Causal Diagrams with DAGs. When you cannot randomize, you have to state your assumptions about what causes what, out loud and on paper. You will learn to draw those assumptions as a diagram and then read straight off it which variables you must adjust for, and which ones you must leave well alone.