Permutation and randomization tests

Today let's understand permutation and randomization tests, a way to get a p-value by actually counting instead of looking one up in a table.

Say an onboarding team is testing a new welcome email against the old one. They measure how many minutes each user is active on the product in their first week. A small pilot sends the new email to 3 users and the old email to another 3. The new email's 3 users log 40, 50 and 60 minutes. The old email's 3 users log 10, 20 and 30 minutes. The new email's average comes to 50 minutes, the old email's to 20 minutes, a gap of 30 minutes.

That 30-minute gap could mean the new email genuinely works better. Or it could be nothing more than how these particular 6 users happened to land, 3 and 3, into the two groups. A permutation test settles that by counting how often a gap this big turns up when the only thing separating the two groups is a random split of the same 6 numbers.

The chart below is the kind of picture that count builds toward.

Its x-axis is standardized, meaning it is measured in how many standard deviations a result sits from zero, so one curve can stand in for a gap of any size or unit. Drag the marker and watch the shaded tail shrink as it moves away from zero. That shaded share is exactly what a p-value measures: how much of the chance-alone curve lies as far out, or further, than the result being judged.

The onboarding pilot, the null, and its 20 possible relabelings

Before trusting that 30-minute gap, state precisely what it is being tested against. The claim being tested is called the null hypothesis, written H0: the new email made no real difference, so which 3 of the 6 users happened to get it is beside the point.

If H0 is true, the "new" and "old" labels are just stickers stuck on 6 fixed numbers, and any other way of handing 3 of those stickers to "new" was just as likely as the one that actually happened.

Press Run to lay out the pilot and check the real gap in R.

RInteractive R
# Lay out the pilot's six users and compute each email's average minutes active pilot <- data.frame( user = 1:6, minutes = c(40, 50, 60, 10, 20, 30), email = rep(c("new", "old"), each = 3) ) pilot #> user minutes email #> 1 1 40 new #> 2 2 50 new #> 3 3 60 new #> 4 4 10 old #> 5 5 20 old #> 6 6 30 old means <- tapply(pilot$minutes, pilot$email, mean) means #> new old #> 50 20 gap <- means[["new"]] - means[["old"]] gap #> [1] 30

  

tapply() splits minutes by email and applies mean() to each group, so means holds the new email's average and the old email's average from one call. gap is just their difference, the same 30 minutes as a moment ago.

Now count every other way those 6 stickers could have landed. Choosing which 3 of the 6 users get the "new" sticker is a combinations problem, and combn(6, 3) lists every one of those choices as a column.

RInteractive R
# List every way to choose 3 of the 6 users for the "new" label combos <- combn(6, 3) dim(combos) #> [1] 3 20 combos[, 1:5] #> [,1] [,2] [,3] [,4] [,5] #> [1,] 1 1 1 1 1 #> [2,] 2 2 2 2 3 #> [3,] 3 4 5 6 4

  

20 columns, each one a different set of 3 user numbers. The column 1 2 3 is the real split, users 1 to 3 getting "new". Every other column is a relabeling nobody actually ran, but one H0 says was exactly as likely as the real one.