Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 1 of 13

Robust Regression with M-Estimators

A rental agency has 12 tidy listings where monthly rent rises neatly with floor area. Then a clerk types one flat's rent as 300 euros instead of 1300, and the whole trend line lurches. Ordinary least squares trusts every row equally, so a single typo can quietly steer the entire model.

Robust regression fixes this by learning which rows to believe. Toggle the buttons below: watch the least-squares line (OLS) get dragged down by the red outlier, then switch to a robust fit and see it snap back to the honest trend while the bad point shrinks to the size of its real influence.

By the end of this lesson you will be able to:

  • Explain why squaring residuals lets a single outlier dominate an OLS fit
  • Tell a high-leverage point apart from an influential one, and measure influence with Cook's distance
  • Fit a robust regression in R with rlm(), read the per-row weights, and know how Huber and Tukey differ

Prerequisites: you can fit and read a simple linear regression with lm(), and you know a residual is the gap between the actual value and the prediction. If lm() is rusty, see linear regression.

The problem

One typo, and the line collapses

Let us rebuild the agency's data so you can see the damage yourself. Twelve clean listings follow a real trend, rent climbing about 11.8 euros for every extra square metre. Then we add the mistyped flat: a 78 square metre unit whose 1300 euro rent was entered as 300. Each lesson runs in a fresh R session, so we build the data right here.

RInteractive R
set.seed(1) size <- c(30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85) # floor area, square metres rent <- round(250 + 11 * size + rnorm(12, 0, 45)) # monthly rent, euros apartments <- data.frame(size, rent) # the 13th listing: a 1300-euro flat mistyped as 300 apartments <- rbind(apartments, data.frame(size = 78, rent = 300)) round(coef(lm(rent ~ size, data = apartments)), 2) # fit WITH the typo #> (Intercept) size #> 389.46 7.78 round(coef(lm(rent ~ size, data = apartments[1:12, ])), 2) # fit WITHOUT it #> (Intercept) size #> 217.97 11.77

  

  





One row knocks the slope from 11.77 down to 7.78, roughly a third. Why is a single point so powerful? Look at what OLS is minimizing. For each apartment \(i\), the residual \(r_i = y_i - \hat{y}_i\) is the gap between the actual rent \(y_i\) and the line's prediction \(\hat{y}_i\). OLS chooses the slope and intercept that make the sum of squared residuals \(\sum_i r_i^2\) as small as possible:



\[ \hat{\beta}^{\text{OLS}} = \arg\min_{\beta} \sum_{i=1}^{n} r_i^2. \]



The squaring is the culprit. The typo sits about 900 euros below the honest line, so it alone contributes roughly \(900^2 = 810{,}000\) to that sum, dwarfing every well-behaved row. To shave down one enormous squared term, the line bends toward the bad point and betrays the twelve good ones. Every row gets full, equal trust, so the loudest mistake wins.