Cluster and Switchback Experiments
Lesson 3 closed on the one pitfall with no statistical fix: interference. Asha saw the referral banner, sent Rohan a coupon, and a control member spent treatment money while every automated check stayed green. You cannot correct that at analysis time. You have to design it away.
This lesson is about the two designs that do it, and they share one move: stop randomizing the individual, and randomize something bigger, so the units that interfere always land on the same side of the split. A cluster experiment fences in space: whole cities, stores or schools. A switchback fences in time: the whole market flips between arms on a randomized schedule. Both work. Both also send you a bill, priced by one number you will learn to compute: the design effect.
By the end of this lesson you will be able to:
- Explain how cluster and switchback designs contain interference, and pick the right fence for a given leak
- Compute the intraclass correlation and the design effect, and turn a headcount into its honest, effective sample size
- Show by simulation why analyzing members when you randomized cities manufactures false winners, and run the correct cluster-level test
- Analyze a switchback at the block level and tighten it by removing the time-of-day pattern
Prerequisites: Lessons 1 to 3 of this course (Designing Experiments for Power, Variance Reduction with CUPED, Experiment Pitfalls, Peeking and SRM) for power, the regression-adjustment trick and the interference problem itself; plus the two-sample comparison (Comparing groups with t-tests).
Randomize the city, not the member
Back to the test that failed. Meera's referral program, give 15 dollars, get 15 dollars, is meant for the shoe store's 40,000 loyalty members, and the member-level version leaked: treated members sent coupons to control friends, control spend rose, and the measured lift shrank. Now use the one thing the leak told us: coupons travel along friendships, and friends overwhelmingly live in the same city.
The loyalty program operates in 40 cities, about 1,000 members each. So instead of assigning each member a coin flip, Meera flips 40 coins, one per city: in 20 cities every member sees the referral banner, in the other 20 nobody does. When Asha sends Rohan a coupon now, it does not matter, because Asha and Rohan both live in Jaipur, and all of Jaipur is in the same arm. The coupon lands inside the fence.
Two pieces of vocabulary make the design precise:
- The unit of randomization is what the coin flip assigns: here, the city. This is called a cluster-randomized experiment, and the city is the cluster.
- The unit of measurement can stay small: Meera still records spend per member. Randomize big, measure small.
One honest caveat before the enthusiasm: the fence only contains what actually stays inside it. Cluster randomization assumes interference is mostly within clusters, friends in the same city, shoppers in the same store, pupils in the same school. A coupon mailed from Jaipur to a cousin in Surat still leaks. Pick the cluster to match the plumbing you mapped in Lesson 3: the point of the exercise is that most sharing happens inside the boundary you randomize.