Choosing interactions before you look

Today let's understand why hunting for interactions in a regression model is riskier than it looks.

Say you are building a regression model with ten candidate predictors, x1 through x10. Nothing stops you from adding a term for how two of them might work together, called an interaction, where one predictor's effect on the outcome depends on the value of another. The trouble starts the moment you ask how many such pairs even exist.

Ten predictors offer forty-five possible two-way pairings. Test every single one against the usual 5% threshold, and something is going to come back looking real, whether or not anything is actually there.

That is the shape of the problem laid out above: a lot of candidate pairs, a search through all of them, and a result that looks like a discovery even when it is not one.

Ten predictors, forty-five interactions

Before testing anything, it helps to see exactly where that number forty-five comes from.

With p predictors, the number of distinct pairs you can form is p(p-1)/2. Order does not matter here: x1 paired with x2 is the same pair as x2 paired with x1, so each pair is only counted once. For 10 predictors that works out to 10*9/2 = 45.

Here is the dataset itself: 300 observations on 10 predictors named x1 through x10, with an outcome y built to have no relationship to any of them. Press Run to build it and confirm the count of pairs.

RInteractive R
# Build the running noise dataset: 300 observations, 10 predictors unrelated to y, and count the possible interaction pairs set.seed(42) n <- 300 p <- 10 X <- matrix(rnorm(n * p), nrow = n, ncol = p) colnames(X) <- paste0("x", 1:p) y <- rnorm(n) noise_data <- data.frame(y = y, X) choose(10, 2) #> [1] 45

  

rnorm(n * p) draws 3,000 independent values from a standard normal distribution and matrix() arranges them into 300 rows and 10 columns, one column per predictor. y is drawn the same way, completely separately from X, so nothing links any predictor to the outcome. Every true main effect and every true interaction in this dataset is exactly zero, by construction. That matters for everything that follows: whatever looks significant later on came from chance alone, because there is genuinely nothing here to find.