A/B Testing and Experiment Design
In Lesson 2 you learned to draw a causal diagram and adjust for confounders because you could not randomize. This lesson is about the happier case: when you can run the experiment. Randomizing is the single move that erases every backdoor at once, collapsing a tangled DAG back to the one clean arrow you care about.
Riverside Books wants to add a "Readers also bought" panel to each book page, hoping it nudges more visitors to buy. Instead of guessing, they will run an A/B test: show the old page to a random half of visitors and the new panel to the other half, then compare buy rates. But a real experiment raises hard, practical questions. How large a difference counts as real, and not just a lucky week? How many visitors do we need before we can trust the answer? The picture below is the tool that settles the first question, and by the end you will size the experiment to earn a trustworthy one.
By the end of this lesson you will be able to:
- Explain why randomly assigning visitors lets a plain comparison of two groups estimate a true causal effect
- State the hypotheses of an A/B test and read a p-value and significance level correctly, without the usual traps
- Define statistical power, and compute the sample size an experiment actually needs in R
- Spot the design mistakes (peeking, chasing many metrics, broken randomization) that quietly break a test
Prerequisites: you finished Lesson 1 (randomization removes selection bias) and Lesson 2 (backdoor paths), and you can run R and compute a proportion. Every new term is defined as it appears.
An A/B test is just a randomized experiment
An A/B test is a randomized controlled experiment wearing a product hat. You take one change you are unsure about, split your users into two groups by chance, expose each group to one version, and compare an outcome. Group A is the control: Riverside's current book page. Group B is the treatment: the same page plus the "Readers also bought" panel. Everything else is held identical.
The outcome Riverside will measure is the conversion rate: the fraction of visitors who buy at least one book. Write \(p_A\) for the true conversion rate a visitor would have under the old page and \(p_B\) for the rate under the new panel. The quantity the whole experiment exists to estimate is the lift,
\[ \Delta = p_B - p_A, \]
the extra share of visitors who buy because of the panel. We never see \(p_A\) or \(p_B\) directly; we estimate them from counts. If \(x_A\) of \(n_A\) control visitors buy, the observed rate is \(\hat p_A = x_A / n_A\), and likewise \(\hat p_B = x_B / n_B\) for the treatment. The hat means "estimated from the sample."
Why go to the trouble of a coin flip? Because of exactly the lesson from before. If Riverside instead showed the panel to whoever happened to visit on Tuesdays, or to logged-in members only, the two groups would differ in ways beyond the panel, and the gap in buy rates would blend the panel's effect with those differences: selection bias, all over again. Randomizing severs the link between which version you see and everything else about you, so the two groups are comparable and the difference in their rates is the panel's doing. Here is the whole procedure.