When You Cannot Randomize
In Lesson 4 you turned Riverside Books' A/B test into an honest, bounded conclusion. That test was clean because Riverside got to flip a coin: each visitor was randomly shown the old page or the new panel, so the two groups were alike in every way except the one thing being tested. Randomization is what made the causal claim easy.
Most real questions do not come with a coin flip. The change already happened before anyone thought to test it. Or randomizing would be impossible, unfair, or absurd: you cannot randomly assign customers to "loves reading" or "does not," and you cannot give half your city free shipping while charging the other half just to measure it. When you cannot randomize, you are left with data the world handed you, and a hard job: recover a causal effect anyway, honestly.
This lesson gives you the two workhorse tools for exactly that, and, just as importantly, teaches you to name the price each one charges. Every method below buys a causal answer with one assumption you can never fully prove. The map below is the whole lesson.
By the end of this lesson you will be able to:
- Explain why comparing two groups that chose themselves is biased, and measure that bias against the truth
- Build a matched comparison group in R, and state the assumption it rests on
- Compute a difference-in-differences estimate, read the counterfactual it draws, and state the assumption it rests on
- Pick a method by the assumption you can actually defend, and scope your causal claim honestly
Prerequisites: you finished Lesson 1 (confounding, potential outcomes \(Y(0)\) and \(Y(1)\), what a causal effect is) and Lessons 3 to 4 (a randomized A/B test, the confidence interval). You can run R and take a mean. Every new term is defined as it appears.
Comparing groups that chose themselves
Here is Riverside's new problem. It launched a free membership (early access to sales, a nicer wish-list), and the numbers look great: members spend far more than non-members. Marketing wants to credit the membership. But nobody was assigned to join. Customers who signed up were the keen ones, the people who already spent more. So the two groups were never alike, and the raw gap mixes two things: any real effect of the membership, plus the head start the joiners already had.
We can see the trap exactly, because we will build the data ourselves and secretly plant a known truth. Each customer has a prior (last year's spend) and a spend (this year's). We set the real membership effect to exactly $5, and let members be the higher-prior customers to begin with. Each lesson runs in its own fresh R session, so we build everything inline.
The naive gap is $14.75, almost three times the true $5. Written as a formula, the comparison you were tempted to make is not the effect alone:
\[ \underbrace{\bar Y_{\text{members}} - \bar Y_{\text{non-members}}}_{\text{naive gap } = \$14.75} \;=\; \underbrace{\tau}_{\text{true effect } = \$5} \;+\; \underbrace{\text{selection bias}}_{\text{joiners already spent more}}, \]
where \(\bar Y\) is a group's average spend and \(\tau\) (tau) is the causal effect we want. The extra $9.75 is selection bias: the difference in what the two groups would have spent even with no membership at all. The picture makes the head start obvious.