Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 3 of 7

Conformal Prediction for Classification

Lesson 2 left Rohan the real-estate agent with a band whose coverage was guaranteed but whose width was rigid. One fixed half-width, applied to every home, covered 90% of listings overall, yet only about 81% of his big, high-scatter homes and a wasteful 98% of the small ones. A guarantee about the average that quietly fails the very clients who matter most.

This lesson makes conformal prediction adapt. First we let the band's width follow the data, so it stretches over the uncertain homes and tightens over the easy ones, rescuing that under-covered subgroup. Then we carry the whole idea across a bridge that surprises most people: to classification, where "adapt" means the model returns not one label but a set of labels, a set that stays small when it is confident and grows only when it genuinely hesitates.

By the end of this lesson you will be able to:

  • Build a conformalized quantile regression (CQR) band whose width adapts to the input, and watch it equalize coverage across subgroups
  • Turn a classifier's probabilities into a conformal prediction set, and explain exactly why the set grows when the model is unsure
  • Measure a set predictor's coverage and average set size, and fix a class that the average is quietly starving (class-conditional, or Mondrian, coverage)

Prerequisites: Lesson 2 (calibration set, nonconformity score, the conformal quantile, and why coverage is marginal not conditional) and Lesson 1 (prediction intervals and coverage). Basic R: vectors, logical indexing, sort, and writing a small function. Every new term is defined as it appears.

Where we left off

One width cannot fit every home

Recall the shape of Rohan's market: price climbs with size at roughly \$180 a square foot, but the scatter grows with size too, so a 2,500 sq ft home is far less predictable than a 700 sq ft one. Let us rebuild it and run Lesson 2's constant-width band on it once more, to see the failure with our own eyes. Each lesson runs in a fresh R session, so we create the data right here (run this once).

RInteractive R
set.seed(5) N <- 3000 sqft <- round(runif(N, 600, 2600)) price <- round(60000 + 180 * sqft + rnorm(N, 0, 18 * sqft)) # spread grows with size homes <- data.frame(sqft, price) train <- homes[1:1000, ] # fit the model calib <- homes[1001:2000, ] # measure its misses test <- homes[2001:3000, ] # final coverage check big <- test$sqft > 1600 # the large, high-scatter homes sum(big) #> [1] 516

  

  





Now the Lesson 2 recipe exactly: fit a model, score the calibration misses by their absolute residual, take the conformal quantile as one half-width \(\hat q\), and wrap \(\hat y \pm \hat q\) around every home. Watch how a single width lands on the big homes versus the small ones.




  

    
RInteractive R

    
fit      <- lm(price ~ sqft, data = train)
resid    <- abs(calib$price - predict(fit, calib))          # nonconformity = |residual|
qc       <- sort(resid)[ceiling((nrow(calib) + 1) * 0.90)]  # ONE fixed half-width
guess    <- predict(fit, test)
in_const <- test$price >= guess - qc & test$price <= guess + qc
round(c(overall = mean(in_const),
        big     = mean(in_const[big]),
        small   = mean(in_const[!big])), 3)
#> overall     big   small
#>   0.884   0.795   0.979

    

      
      
    

    

  

  





There it is again, in cold numbers. The overall coverage is a healthy 88%, but that average is a blend of 79.5% on the big homes and 97.9% on the small ones. The one width Rohan can offer is too tight where the market is wild and far too loose where it is calm. The band needs to breathe.