Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 5 of 13

Lasso and Elastic Net

Last lesson, ridge regression rescued Maya's marketing model. Her six ad channels moved almost in lockstep, and ordinary least squares had responded with nonsense: coefficients that swapped, went negative, and lurched every refit. Ridge steadied all of it by shrinking the coefficients toward zero. But it left one question hanging: ridge keeps every predictor. It never tells Maya WHICH channels actually matter.

That question has grown teeth. Maya's weekly export now carries twenty columns, not six: the six ad channels plus fourteen "vanity metrics" her dashboard spits out (newsletter opens, app pings, and a dozen more) that she suspects are pure noise. She does not want twenty tamed coefficients. She wants a short list: the handful of columns worth acting on.

Lasso regression gives her exactly that. With one change to the penalty, it drives useless coefficients to exactly zero, so it does feature selection for free. Elastic net then blends lasso with ridge to handle the correlated channels gracefully. Drag the penalty slider below to watch coefficients shrink and, on the Lasso setting, drop out one by one.

By the end of this lesson you will be able to:

  • Explain why ridge can never hand you a shortlist of predictors, and why lasso can
  • Define the lasso (L1) penalty and say, through soft-thresholding, why it produces exactly zero where ridge only shrinks
  • Fit lasso in R, choose the penalty by cross-validation, and read which predictors survived
  • Explain elastic net, the alpha mixing dial, and the grouping effect that keeps correlated predictors together
  • Choose between ridge, lasso, and elastic net for the problem in front of you

Prerequisites: Ridge Regression and Shrinkage (Lesson 4) is the direct prequel, so you know the L2 penalty, glmnet's alpha argument, and choosing the penalty with cv.glmnet. You can read a linear regression (a coefficient is a predictor's effect; a residual is actual minus predicted), and you have met the bias-variance tradeoff.

The setup

Twenty columns, and most move together

Let us rebuild Maya's data, now wider. Sixty weeks; the same six ad channels that all scale from one weekly budget (so they rise and fall together); plus fourteen dashboard "vanity metrics" that are pure random noise, unrelated to signups. As before, signups truly depend on just two channels, search and social. (We build the data right here, since each lesson runs in a fresh R session.)

RInteractive R
set.seed(1) n <- 60 budget <- rnorm(n) # the one weekly budget dial spend <- sapply(1:6, function(j) budget + rnorm(n, sd = 0.15)) colnames(spend) <- c("search", "social", "display", "video", "email", "affiliate") metrics <- matrix(rnorm(n * 14), n, 14) # 14 dashboard vanity metrics: pure noise colnames(metrics) <- paste0("metric", 1:14) signups <- 40 + 5 * spend[, "search"] + 3 * spend[, "social"] + rnorm(n, sd = 2) ads <- data.frame(signups, spend, metrics) dim(ads) # 60 weeks; signups + 20 predictors #> [1] 60 21

  

  





Two things about this table matter for everything below. First, most of the twenty predictors are junk: the fourteen metrics have no effect on signups at all, and even among the ad channels only search and social carry real weight. A good method should hand Maya a short list. Second, the six ad channels are almost the same column, each pair correlated around 0.97, the near-solid green block in the heatmap. That tangle, called multicollinearity, is what wrecked ordinary least squares in Lesson 4, and it will trip up lasso too, in a subtler way we will fix at the end.