Paying a penalty to get a better model

Today let's understand how ridge and lasso regression trade a little bias for a large cut in variance, and why that usually wins once you are predicting on apartments you have not seen yet.

Let's say you are a real-estate analyst in a mid-size city. You have sale records for 50 apartments that sold recently, and for each one you have 12 measured features: size in square metres, distance from the city centre in kilometres, and number of bedrooms, the 3 that genuinely move the price, plus 9 more columns that a real dataset tends to pick up along the way, columns that look plausible but carry no real relationship to price at all.

Fit an ordinary linear regression, OLS for "ordinary least squares", on all 50 apartments using all 12 columns, and look at what comes back.

RInteractive R
# Build the 50-apartment data (3 real features, 9 noise features), fit OLS on all 12 and print the coefficients set.seed(42) n <- 50 size_m2 <- round(runif(n, 40, 160)) distance_km <- round(runif(n, 0.5, 15), 1) bedrooms <- sample(1:4, n, replace = TRUE) noise_mat <- matrix(round(rnorm(n * 9), 2), nrow = n, ncol = 9) colnames(noise_mat) <- paste0("noise_", 1:9) price <- 50 + 1.8 * size_m2 - 4 * distance_km + 10 * bedrooms + rnorm(n, 0, 15) price <- round(price, 1) apt <- data.frame(price, size_m2, distance_km, bedrooms, noise_mat) fit_ols <- lm(price ~ ., data = apt) coefs <- coef(fit_ols)[-1] round(coefs, 2) #> size_m2 distance_km bedrooms noise_1 noise_2 noise_3 #> 1.82 -4.92 9.33 1.53 -2.81 1.44 #> noise_4 noise_5 noise_6 noise_7 noise_8 noise_9 #> 0.40 -0.55 -0.85 0.55 0.17 -1.57

  

Here are those same 12 numbers again, ranked from largest to smallest by size, with each bar's actual signed coefficient written next to it.

bedrooms and distance_km, 2 of the 3 real features, take the top two spots, no surprise there. But sitting right behind them, in third place, is noise_2 at -2.81, a column with nothing to do with price, outranking even size_m2, one of the real features.

That ranking is the whole problem in one picture. A column with nothing to do with price has picked up a coefficient you would not shrug off if you saw it in a report.

Why more predictors can make a fit less trustworthy

The fit you just saw used all 50 apartments at once. But any real sale sample is really just one particular draw of buyers and sellers. Had a different 50 apartments sold that month, the numbers would come out differently. So split your 50 apartments into two halves and refit the same model on each half separately, and see which coefficients hold up.

RInteractive R
# Refit OLS separately on two random 25-apartment halves and compare the 12 coefficients set.seed(42) idx <- sample(1:n) half1 <- apt[idx[1:25], ] half2 <- apt[idx[26:50], ] fit_half1 <- lm(price ~ ., data = half1) fit_half2 <- lm(price ~ ., data = half2) round(cbind(half1 = coef(fit_half1)[-1], half2 = coef(fit_half2)[-1]), 2) #> half1 half2 #> size_m2 1.75 1.79 #> distance_km -4.26 -5.62 #> bedrooms 10.80 5.67 #> noise_1 5.89 -0.41 #> noise_2 3.95 -2.99 #> noise_3 -0.54 -1.63 #> noise_4 -6.21 3.55 #> noise_5 5.27 -2.25 #> noise_6 1.39 -4.09 #> noise_7 -6.35 4.85 #> noise_8 10.64 -3.97 #> noise_9 -3.76 -1.01

  

Read the first 3 rows first. size_m2 stays close, 1.75 against 1.79. distance_km stays negative in both halves, -4.26 against -5.62. bedrooms stays positive in both, 10.80 against 5.67, even though it moves more than the other two.

Now read the 9 rows below them. 7 of the 9 noise coefficients flip sign between the two halves. noise_8 is the most dramatic: plus 10.64 in the first half, almost as large as that same half's own bedrooms coefficient of 10.80, and minus 3.97 in the second half.

A coefficient that size, on a column with no real relationship to price, is not a fluke of one bad half. It is what happens whenever you ask 25 or 50 rows of data to pin down 12 coefficients at once: some of that noise ends up looking like signal, and which noise looks like signal changes completely depending on which apartments you happened to sample.

This is the bias-variance trade, and it is worth defining precisely. An estimator is biased if, averaged over many samples, it misses the true value. OLS is not biased in that sense: averaged over endless different 50-apartment samples, its coefficients would land on the truth. But an estimator can be unbiased on average and still be a poor estimator on any one sample, because its variance, how much it swings from sample to sample, is too large, and that is exactly what the noise coefficients just showed you: correct on average, useless on this one sample.