Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 4 of 7

Quantile and Distributional Regression

In Lesson 3 you borrowed quantile regression to price Rohan the real-estate agent's homes: you fit a low line and a high line, used the gap between them as an adaptive band, and then wrapped conformal around it to make the coverage a guarantee. Quantile regression was a supporting actor. Now it takes the lead.

This lesson studies it in its own right, and then pushes past it. First, how one small change to ordinary regression, the pinball loss, aims a line at any percentile you like, so a pair of lines becomes a prediction interval whose width breathes with the data. Then the bigger idea the title promises: instead of two or three lines, fit a whole fan of them and you have estimated the entire distribution of the outcome at each input, not just its average. That is distributional regression, and it lets you read any interval, the spread, or the chance of clearing a threshold, straight off one model.

By the end of this lesson you will be able to:

  • Explain how the pinball (check) loss fits a conditional quantile, and why a pair of quantile lines gives a prediction interval whose width adapts to the input
  • Fit conditional quantiles in R with rq(), build an adaptive 90% band, and measure its coverage where a fixed-width interval failed
  • Estimate the whole conditional distribution with a dense grid of quantiles, and read any interval, the conditional spread, or a tail probability off it, while knowing exactly where the method breaks

Prerequisites: Lesson 1 (prediction intervals, coverage, and why one lm width under-covers big homes) and Lessons 2 to 3 (the conformal guarantee, and quantile regression met there as a tool). Basic R: vectors, logical indexing, lm(), data.frame, and writing a small function. Every new term is defined as it appears.

Where we left off

One width, or the whole shape?

Rohan the real-estate agent has the same market as always: price climbs with size at about \$180 a square foot, but the scatter grows with size too, so a 2,500 sq ft home is far harder to price than a 700 sq ft one. Lesson 1 showed the damage this does to an ordinary lm prediction interval: one pooled width, too tight for the big homes and too loose for the small ones. Let us rebuild the market and see that crack one more time, because fixing it is the whole point of what follows. Each lesson runs in a fresh R session, so we create the data right here (run this once).

RInteractive R
set.seed(5) N <- 3000 sqft <- round(runif(N, 600, 2600)) price <- round(60000 + 180 * sqft + rnorm(N, 0, 18 * sqft)) # spread grows with size homes <- data.frame(sqft, price) train <- homes[1:2000, ] # fit the models test <- homes[2001:3000, ] # honest coverage check big <- test$sqft > 1600 # the large, high-scatter homes fit_lm <- lm(price ~ sqft, data = train) band_lm <- predict(fit_lm, test, interval = "prediction", level = 0.90) in_lm <- test$price >= band_lm[, "lwr"] & test$price <= band_lm[, "upr"] round(c(overall = mean(in_lm), big = mean(in_lm[big]), small = mean(in_lm[!big])), 3) #> overall big small #> 0.892 0.810 0.979

  

  





The same story as Lesson 1, in fresh numbers: 89% overall looks fine, but that average blends 81.0% on the big homes with 97.9% on the small ones. The reason is that lm builds its interval from a single pooled error size, so it has exactly one width to give, and that width is nearly identical whether the home is small or large:




  

    
RInteractive R

    
blm <- predict(fit_lm, data.frame(sqft = c(800, 2400)),
               interval = "prediction", level = 0.90)
round(cbind(blm, width = blm[, "upr"] - blm[, "lwr"]))
#>      fit    lwr    upr  width
#> 1 204684 152953 256416 103463
#> 2 492642 440912 544373 103462

    

      
      
    

    

  

  





About \$103,000 wide for the small home and \$103,000 wide for the big one. That rigidity is the enemy. What Rohan needs is a band that is narrow where the market is calm and wide where it is wild. To get one, we stop predicting the middle and padding it, and instead predict the edges directly.