Experiment Pitfalls, Peeking and SRM

Lesson 2 closed with a warning: decide the analysis before the results exist. This lesson is about what happens when you do not, and about two more ways a perfectly sized, CUPED-tightened experiment can still hand you a wrong answer.

Meera's one-page checkout test from Lesson 1 is finally live: 4.0% baseline conversion, a hoped-for 4.6%, 17,943 visitors per arm, six weeks. Every morning her dashboard recomputes the significance test on the traffic so far. On day 9 of 42 it shows p = 0.04 with a green banner: significant. Her product manager asks the obvious question: why wait five more weeks?

This lesson covers the three ways online experiments lie: peeking (that green banner), sample-ratio mismatch (a broken randomizer or logging pipeline deleting sessions from one arm), and interference (users leaking treatment onto each other). By the end you will be able to:

• Explain why checking a running test repeatedly inflates false winners, and measure the inflation by simulation
• Decide what to do when a mid-test p-value dips under 0.05, and name the honest ways to look early
• Run a sample-ratio-mismatch check with a chi-square test and read it at the 0.001 bar
• Say why even a small mismatch invalidates the result, and which way interference bends an estimate

Prerequisites: Lessons 1 and 2 of this course (Designing Experiments for Power, Variance Reduction with CUPED) for power, the planned sample size and the analysis-plan discipline; plus the p-value and the two-proportion comparison (Reading an Experiment). The chi-square test appears here as a tool and is explained where it lands; Chi-Square Goodness of Fit Test in R has the full treatment.

Below is the machinery behind Meera's dashboard: where the test statistic lands when the redesign does nothing, with the p-value as the shaded tail. Drag the slider: at z = 1.75 the banner stays grey, at 1.96 it turns green. Keep that flip in mind. The whole first pitfall is how often a NULL experiment wanders across it when you hand it 42 chances.

A winner every morning, if you wait for one

Start with what the dashboard actually did. Day 9: p = 0.04, green. Day 10: p = 0.09, grey again. Day 11: p = 0.13. Nothing broke. With only 3,870 visitors per arm so far, the two conversion estimates wobble, and the p-value wobbles with them. The trouble is not that Meera looked; it is what a look invites: stop the moment the banner turns green. Statisticians call this peeking, or optional stopping, and it is the most common way honest teams manufacture false winners.

Recall the deal from Lesson 1. Setting \(\alpha = 0.05\) buys exactly this promise: if the redesign does nothing, a significance test run ONCE, at the planned sample size, flags a false winner 5% of the time. The promise says nothing about a test recomputed every morning. Each look is a fresh chance for a random wobble to cross the bar, and the stop rule is one-sided: Meera stops when the banner is green, never when it turns grey again the next day. The data get 42 attempts; the null hypothesis gets none.

How bad can it get? If the \(k\) looks were independent tests, the chance that at least one dips under \(\alpha\) would be

\[ P(\text{at least one false winner in } k \text{ looks}) = 1 - (1 - \alpha)^k \]

where \(k\) is the number of looks and \(\alpha\) is the per-look significance level. Read the growth:

Daily looks are not independent, though: day 10's test contains every visitor from day 9, so consecutive p-values are strongly correlated, and the true inflation is smaller than this table claims. How much smaller is an empirical question, and you already own the right tool for it: simulate. One thing, however, holds exactly: keep a null test running and peeking indefinitely and the false-winner rate keeps climbing toward certainty, a classical result proved by Armitage, McPherson and Rowe back in 1969.