Why stepwise selection lies to you
Today let's understand why a model chosen by an automatic search can look real even when there is nothing real in the data behind it.
Say you are fitting a regression model and are not sure which of many candidate predictors actually belong in it. A common shortcut is to let an algorithm decide for you: start with every predictor in one model, then repeatedly drop whichever one helps the fit most when removed, stopping once no drop helps any further. This is called stepwise selection, specifically backward elimination, and the usual pitch is that whatever predictors remain at the end earned their place.
Here is the test this lesson runs on that pitch. We are going to build 40 predictor columns that have, by construction, no relationship at all to the outcome. If stepwise selection does what it's pitched to do, running it on this data should hand back close to nothing.
The search itself is just four steps, repeated until nothing changes.
Start with everything, drop whatever helps most, repeat until nothing helps, then report whoever is left standing. That is the whole procedure, about to run on 40 predictors built to contain nothing at all.
A backward search run on pure noise
Before running the search, here is what it is actually optimizing. AIC stands for Akaike Information Criterion, and it scores a model by weighing fit against predictor count: every added predictor charges the score a fixed 2 points, so a predictor only earns its place if it improves the fit by more than that. Lower AIC is better.
Backward elimination starts from the full model. At each round it drops whichever predictor lowers the AIC the most, and it stops the moment every remaining removal would raise the AIC instead.
Here is the data the search will run on: 100 observations, 40 predictor columns named x1 through x40, every one of them independent random noise with no built-in connection to the outcome y, which is random noise too. Fit the full model, then let backward elimination run on it.
Seven predictors survived: x1, x8, x17, x18, x19, x22, and x34. The AIC fell from 312.77 on the full 40-predictor model to 266.19 on this seven-predictor one, which is exactly why the other 33 went home: dropping each of them lowered the AIC further, while dropping any of these seven would have raised it.
Now read the bottom of that printout. R-squared is 0.2546, so this model explains about a quarter of the variation in y. The overall F-test comes back at F = 4.489 on 7 and 92 degrees of freedom, with a p-value of 0.000245, comfortably below the usual 0.05 cutoff.
Every one of those 40 predictors was built with zero relationship to y. There is nothing in this data for any model to find. And yet the search returned one that, by every number on the page, looks real.
A p-value computed as though no search happened
The predictor x8 came back with a t-value of 2.81 and a reported p-value of 0.006, the smallest and most convincing-looking of the seven survivors.
That p-value is computed as though x8 were the one predictor decided on in advance, the single test somebody had planned to run before seeing any data. But x8 was never decided on in advance. It was the strongest performer out of 40 candidates, picked precisely because it looked the best among all of them.
One way to see how much that distinction costs is to picture the distribution a test statistic would follow if nothing at all were going on, and ask how far out in the tail 2.81 really sits.
The slider starts at 2.81, x8's own t-value. The shaded tail, the share of a standard normal distribution at least this far from zero in either direction, works out to about 0.005. That is close to R's own 0.006 but not identical, because R's exact figure comes from a t-distribution with 92 degrees of freedom rather than a standard normal; the two agree closely at this sample size without being the same number.
Here is that raw p-value, read directly off the model.
That number only holds if x8 were the one predictor ever tested. The honest version treats all 40 candidates as having been tested, and the simplest way to do that is a Bonferroni correction, which multiplies the p-value by the number of tests performed.
Once 40 tests are accounted for, x8's p-value rises from 0.006 to 0.242. A p-value of 0.242 is nowhere near significant. The predictor that looked like the strongest finding in the whole model was never actually tested the way its reported p-value of 0.006 assumes.
Quick check: what a post-search p-value measures
A predictor survives backward elimination with a reported p-value of 0.006. Which statement reads that number correctly?
Why the surviving coefficients run too large
The p-value is not the only number a search distorts. The size of the coefficients that survive is wrong too, and in a specific direction: too large.
Here is the same construction, run smaller and repeated. Twenty separate noise datasets, each with 100 observations and 20 candidate predictors, every true coefficient exactly zero in every single one of them. Fit the full model and the backward-selected model on each of the 20, and collect two piles of numbers: every coefficient from every full model, and only the ones that survived selection.
Four hundred candidate coefficients went into the first pile, one for every predictor in every one of the 20 full models, and their average size is 0.094. Only 93 coefficients survived selection across all 20 datasets, about 4.65 per dataset, and their average size is 0.194, almost exactly twice as large.
Every one of those coefficients, surviving or not, was estimated on a predictor with a true value of exactly zero. Nothing in the data favors any predictor over any other. A predictor only survives backward elimination because, in this one sample, its estimate happened to come out large, and large estimates are exactly what the search is built to keep. The ones that make it through are a biased, too-large sample pulled from a population that is actually all zeros.
Lasso and ridge: a fixed penalty in place of a hidden one
Stepwise selection is not the only way to decide which predictors stay in a model. A different family of methods, ridge and lasso regression, handles that same decision in a way that is honest about its own cost.
The widget below runs on its own six predictors, three of them genuinely related to the outcome and three of them noise; it is a second, separate worked example, not the 40-predictor data built earlier in this lesson.
Drag the penalty slider, called lambda, and watch the six coefficients as it grows. Under lasso, the weaker coefficients, including the three noise predictors, get driven all the way to exactly zero: that is a form of predictor selection, but one with a stated, explicit cost built into the penalty itself. Under ridge, every coefficient shrinks smoothly toward zero as lambda grows, but none of them ever reaches it exactly.
Either way, the amount of shrinkage is fixed by lambda before the model is ever fit to the data. Nobody tries one set of predictors, checks how it looks, and then tries another. That is the difference that matters: a stepwise search keeps its cost out of the p-values and coefficients it reports, which read as though no search ever happened, while a penalty like lambda is declared up front and paid openly.
What to do instead
Here is the model that is actually honest about this lesson's data.
The null model, with no predictors at all, has an R-squared of exactly 0. That is the truth, because every one of the 40 predictors was built with no relationship to y. Compare that zero against the R-squared of 0.2546 and the p-value of 0.000245 the backward search returned earlier in this lesson. The search did not find a real relationship hiding in the data. It found the best-looking noise out of 40 tries, and reported it using numbers that assume no search happened.
The fix is not a smarter search. It is to stop searching. Decide which predictors belong in the model from what you know about the mechanism connecting them to the outcome, before you ever look at this dataset's own numbers, and then fit that one model once. Whatever degrees of freedom you have available get spent on the predictors you can defend on those grounds, not on whichever ones happened to look best in one sample.
Quiz: reading a stepwise result
A colleague ran backward elimination on 30 candidate predictors, kept 4 of them, each significant at p < 0.05, with an overall R-squared of 0.22. Which conclusion is best supported?
Your turn: run the check yourself
Time to run this check on data of your own, not the lesson's.
Fresh noise, built the same way as before: 80 observations, 15 candidate predictors, every one unrelated to the outcome, with a seed not reused anywhere earlier in this lesson. The full model is already fit for you below.
Show answer
# Complete the search and read the result
step_check <- step(full_check, direction = "backward", trace = 0)
summary(step_check)
#>
#> Call:
#> lm(formula = y ~ x1 + x2 + x6 + x8 + x12, data = noise_check)
#>
#> Residuals:
#> Min 1Q Median 3Q Max
#> -1.8707 -0.6932 0.0359 0.6391 1.9823
#>
#> Coefficients:
#> Estimate Std. Error t value Pr(>|t|)
#> (Intercept) -0.1342 0.1070 -1.254 0.2138
#> x1 0.2138 0.1214 1.761 0.0824 .
#> x2 0.2440 0.1052 2.318 0.0232 *
#> x6 -0.2364 0.1066 -2.217 0.0297 *
#> x8 0.1823 0.1185 1.539 0.1282
#> x12 -0.1970 0.1054 -1.869 0.0655 .
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Residual standard error: 0.926 on 74 degrees of freedom
#> Multiple R-squared: 0.2392, Adjusted R-squared: 0.1878
#> F-statistic: 4.653 on 5 and 74 DF, p-value: 0.0009517Five noise predictors came back looking significant, with an R-squared of 0.239 and an overall p-value under 0.001, from data built to contain no relationship at all. That is the same lie this lesson opened with, only smaller.
References
- A Note on Screening Regression Equations - Freedman, D.A. (1983), The American Statistician, 37(2), 152-155. The original demonstration that screening many predictors and keeping the best-looking ones returns a model that looks significant even when none of the predictors are real.
- Regression Modeling Strategies, course notes - Harrell, F.E. Companion to the book (2nd ed., Springer, 2015); the chapter on variable selection argues for choosing predictors on subject-matter grounds instead of an automatic search, the alternative this lesson closes on.
- Miller, A.J. "Subset Selection in Regression." 2nd ed., Chapman and Hall/CRC (2002). A full account of how automatic subset-selection methods distort the p-values and coefficients that come out of them.
- Flom, P.L. & Cassell, D.L. "Stopping Stepwise Regression." NESUG Proceedings (2007). A practical case against stepwise selection aimed at working analysts rather than theoreticians.
- Why do we still use stepwise modelling in ecology and behaviour? - Whittingham, M.J., Stephens, P.A., Bradbury, R.B. & Freckleton, R.P. (2006), Journal of Animal Ecology, 75(5), 1182-1189. A survey of how often stepwise selection is still used and misread in published research.
The lie, and the honest alternative
Forty predictors, every one of them pure noise, went into a backward AIC search. What came out was a seven-predictor model with an R-squared of 0.2546 and an overall p-value of 0.000245, numbers that would get a result published.
Both of those numbers lie about what actually happened, and in two separate ways. The p-value on the strongest survivor, x8, is computed as though it were the one predictor tested, when it was really the best of 40; correcting it with a Bonferroni adjustment sends it from 0.006 to 0.242, nowhere near significant. And the coefficients that survive any such search run too large: repeated across 20 noise datasets, the coefficients backward elimination kept averaged about twice the size of a typical candidate coefficient before any selection happened, even though the truth, every time, was zero.
The honest alternative is not a better search. Choose which predictors belong in a model from what you already know about the mechanism connecting them to the outcome, before you fit anything, and spend the degrees of freedom you have on the predictors you can defend that way.