Designing Experiments for Power
Meera runs growth at an online shoe store. The checkout page gets about 6,000 visitors a week, and 4.0% of them complete a purchase. Her team has built a one-page checkout they believe will lift that to 4.6%. A 0.6 percentage-point lift sounds tiny, but across 312,000 visitors a year at an $85 average order, it is worth about $160,000 a year.
Before Meera splits any traffic, one question decides whether the test is worth running at all: how many visitors does it need? Too few, and the test will fail to notice a redesign that genuinely works. This lesson is about answering that question before a single visitor is randomized.
By the end you will be able to:
- Name the two ways an A/B test can be wrong, and say what statistical power means
- Estimate a design's power by simulating the experiment before running it
- Compute the sample size a test needs, and the smallest lift your traffic can detect
- Re-plan a test when the expected effect shrinks, using the effect/alpha/n tradeoff
Prerequisites: you have met the A/B test and the p-value (A/B Testing and Experiment Design, Reading an Experiment), and you can write a short function in base R (Comparing groups with t-tests covers the two-sample test we will be sizing).
A test can be wrong in two directions
Here is the experiment Meera plans to run. Half of checkout visitors see the old page, half see the new one-page design. After some number of weeks she compares the two conversion rates with a significance test and reads the p-value: below 0.05, declare the redesign a winner; above, do not ship it.
That verdict can go wrong in exactly two ways, because there are two possible worlds and two possible verdicts:
| Test says significant | Test says not significant | |
|---|---|---|
| The redesign truly lifts conversion | Correct detection, probability \(1-\beta\) (the power) | Miss (Type II error), probability \(\beta\) |
| The redesign does nothing | False winner (Type I error), probability \(\alpha\) | Correct |
The first error you already know. \(\alpha\) (alpha) is the false-alarm rate you chose when you set the significance level: if the redesign does nothing, a 0.05 test hands Meera a false winner 5% of the time. The interactive below is that familiar machinery: the bell curve is what the test statistic does when nothing is going on, and the shaded tail is the p-value. Drag the observed statistic and watch when the verdict flips.
The second error is the one this lesson exists for. Suppose the one-page checkout genuinely lifts conversion from 4.0% to 4.6%, and the test still comes back not significant, just because too few visitors passed through. Meera scraps a $160,000-a-year improvement and never learns she was holding it. That is a Type II error, a miss, and its probability is written \(\beta\) (beta).
Statistical power is the flip side of the miss: \(\text{power} = 1 - \beta\), the probability the test detects the lift, assuming the lift is real and of a specific size. The working convention is to design for at least 80% power, the same way 0.05 is the convention for \(\alpha\). In short: alpha protects you from shipping junk; power protects you from throwing away gold.