Controlling the family-wise error rate

Today let's understand what happens to a familiar 5% risk once you stop running one test and start running twenty.

A crop-science lab is testing a new soil treatment against an untreated control. Instead of measuring one thing, the lab records 20 plant traits on the same plants: height, leaf count, root mass, chlorophyll content, and 16 others. Twelve plots get the treatment, twelve stay untreated, and every trait gets its own t-test at the usual alpha = 0.05.

That is 20 tests and 20 p-values. Here is what they look like, one point per trait, plotted against the 0.05 line each test was judged against.

Five of those twenty points land below the line. Whether that is five traits where the treatment genuinely worked, or something else entirely, is what the data underneath them will decide.

What a 5% risk per test becomes across 20 of them

Start with just one test. Setting alpha to 0.05 means accepting a 5% chance that this one test calls a difference real when none exists.

Now run 20 independent tests, each carrying that same 5% risk on its own. The risk for the family of 20 taken together does not stay at 5%, it compounds. The easiest way to see why is to flip the question around: what is the chance that none of the 20 tests throws a false alarm, then subtract that from 1.

The chance that one test stays quiet is 1 minus alpha, which is 0.95. The chance that all 20 stay quiet, assuming the tests are independent, is 0.95 raised to the 20th power. So the chance that at least one of the 20 throws a false alarm is 1 minus (1 minus alpha) to the power of 20.

Here is that worked out for a few family sizes.

RInteractive R
# How a 5% per-test risk compounds across k independent tests alpha <- 0.05 k <- c(1, 5, 10, 20) fwer <- 1 - (1 - alpha)^k names(fwer) <- paste0("k=", k) round(100 * fwer, 1) #> k=1 k=5 k=10 k=20 #> 5.0 22.6 40.1 64.2

  

At k = 1 the risk is just alpha, 5%. By k = 5 it has already climbed to 22.6%. At k = 10 it is 40.1%. By the time the crop trial's full set of 20 tests is counted, a family that started at a 5% risk per test carries a 64.2% chance that at least one of them throws a false alarm, even if the treatment does nothing at all, anywhere.

This quantity, the chance that a family of tests throws at least one false alarm when none of them should, has its own name: the family-wise error rate (FWER). Controlling it, instead of just the per-test alpha, is exactly what Bonferroni and Holm are built to do.