When false discovery rate beats Bonferroni

Today let's look at a correction built for a different kind of problem: not one confirmatory test, but a screen hunting for as many real effects as it can find among thousands of candidates.

A lab has run a two-sample t-test on each of 1,000 genes, checking whether a new drug shifted how active that gene is. That is 1,000 p-values, one per gene. Here are five of them, picked from the list.

Look at gene 205's p-value, 0.766, next to gene 3's, 0.00018. On their own, five numbers like these do not say which genes the lab should flag as real. Deciding that, out of all 1,000, is exactly what this lesson works out, using two rules that each guarantee something different.

Why testing many things at once multiplies your error rate

Before getting to the lab's full screen, it helps to see why running many tests at once is dangerous in the first place. That danger is exactly what Bonferroni was built to fix.

Say you run one test at the usual alpha of 0.05. If nothing is really going on, there is still a 5% chance that one test comes back significant, purely by chance. Now run k of those tests, each one independent and each one carrying that same 5% risk on its own. The chance that at least one of the k comes back significant, even though none of them are real, is 1 minus the chance that every single one of them stays quiet: 1 - (1 - alpha)^k.

RInteractive R
# How a 5% per-test risk compounds as the number of independent tests grows alpha <- 0.05 k <- c(1, 5, 10, 20, 30, 1000) fwer <- 1 - (1 - alpha)^k names(fwer) <- paste0("k=", k) round(100 * fwer, 1) #> k=1 k=5 k=10 k=20 k=30 k=1000 #> 5.0 22.6 40.1 64.2 78.5 100.0

  

At k = 1 that risk is just alpha, 5%. By k = 20 it has already climbed past 64%. And at k = 1,000, the scale of the lab's actual screen, it is essentially a certainty: some test will cross 0.05 by pure luck, even if the drug does nothing anywhere.

Bonferroni's fix is to make every individual test stricter by exactly the number of tests in the family. Instead of comparing each p-value to alpha, it compares each one to alpha divided by k. Judge the smallest p-value against a bar that strict, and the chance that any of them clears it by pure luck drops back down to about alpha.

The widget below lets you watch that happen. It simulates a world where nothing is real anywhere, exactly the case the formula above describes, and counts how often at least one test still clears the bar. Start with "No correction" and drag k up from 1, then switch to Bonferroni with k left where it is.

With no correction, the share climbs toward the number the formula predicted. Switch to Bonferroni, and it drops back down near alpha, whatever k is set to. The widget's own R code underneath runs the same idea directly: draw k random numbers between 0 and 1 with runif(k), which is exactly what a p-value looks like under a true null, and check whether the smallest one clears the bar.

Note
This widget only models a world where every test is testing a true null, nothing real anywhere. The lab's actual screen is not like that: 200 of its 1,000 genes genuinely respond to the drug. The widget's job is just to make the multiplying danger, and Bonferroni's fix for it, concrete before the real screen arrives next.