Variance Reduction with CUPED

In Lesson 1, Meera hit a wall: at her store's traffic, a small checkout lift needed weeks of collecting data before the test had enough power to see it. This lesson is about the escape hatch. There is a way to make the very same experiment far more sensitive, using data you already collected before the test even started, and it changes nothing about how you randomize.

By the end you will be able to:

• Explain where the noise in an A/B metric comes from, and why data from before the test predicts most of it
• Apply the CUPED adjustment in base R, and report the tighter confidence interval it buys
• Predict the variance cut from a single number (the correlation) and translate it into the sample size you save
• Recognize when the adjustment is invalid or useless, so you never fool yourself with it

Prerequisites: Lesson 1 (Designing Experiments for Power) for power and the standard error of a difference in means; the A/B test and the p-value (A/B Testing and Experiment Design, Reading an Experiment); and a two-sample comparison in a little base R (Comparing groups with t-tests).

Where the noise in your metric comes from

Meera's next test is not about conversion. She wants to know whether a new "Picked for you" shelf lifts spend per member over the test month. She shows it to 4,000 randomly chosen loyalty members, 2,000 per arm, and hopes for about a $2.50 lift per member, worth roughly $100,000 a month across her 40,000-member base.

Here is the trouble. Members are wildly different from each other. Look at three of them, and at what each spent the month before the test:

Priya is a big spender every month; Sam is a small one. That gap between members has nothing to do with the shelf, yet it dwarfs a $2.50 effect. Across all 4,000 members, spend swings by about $39 (one standard deviation) from person to person. It is like trying to hear a whisper in a stadium.

But notice something in the table. For Priya and Sam, this month looks a lot like last month. The member-to-member spread is not random churn; it is stable, and last month's spend already tells us most of it. Only Dana surprises us: normally a small spender, she splurged this month. That leftover surprise is the part we truly cannot predict.