Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 2 of 7

Split Conformal Prediction

Last lesson left Rohan the real-estate agent with a broken promise. His model's tidy "90% prediction interval" quietly covered only about 82% of his biggest listings, because it trusted the errors to be a neat, equal-variance bell curve. The moment that assumption cracked, the 90% on the label became fiction.

This lesson builds a band that makes no such assumption, and that works around any model you hand it, even a bad one. Split conformal prediction reads its width straight off the model's own misses on held-out data, and in return it guarantees its coverage: at least the fraction you asked for, whatever shape the errors take, whatever model produced them, from one held-out slice and a single quantile.

By the end of this lesson you will be able to:

  • Build a conformal band by hand in R: split into train and calibration, score each held-out miss, take the conformal quantile as the half-width, and wrap it around any model
  • Explain why the scores must come from a held-out calibration set, not the model's own training misses, and see coverage collapse when you forget
  • Measure a band's empirical coverage, state exactly what the guarantee does and does not promise, and see why one width cannot cover every subgroup

Prerequisites: Lesson 1 (prediction intervals, coverage, empirical coverage), and the fact that a residual is actual minus predicted (OLS regression, regression assumptions). Basic R: vectors, logical indexing, data.frame, sort. Every new term is defined as it appears.

The one idea

From a bet to a guarantee

Every interval so far has been a bet. Lesson 1's textbook band bet that the errors were a symmetric bell curve, plugged that bet into a formula, and lost when Rohan's biggest homes broke it. Split conformal refuses to bet. Instead it asks one blunt, honest question: on homes the model has never seen, 90% of the time it missed by less than how much? That amount becomes the half-width of the band. No bell curve, no variance formula, nothing assumed about the errors at all.

Let us rebuild Rohan's market to try it on. Each lesson runs in a fresh R session, so we create the data right here (run this once). Price climbs with size at about $180 a square foot, and the noise deliberately grows with size, so big homes scatter far more than small ones (renovations, a bidding war, a dream kitchen).

RInteractive R
set.seed(1) n <- 1200 sqft <- round(runif(n, 600, 2600)) price <- round(60000 + 180 * sqft + rnorm(n, 0, 18 * sqft)) # spread grows with home size homes <- data.frame(sqft, price) head(homes, 4) #> sqft price #> 1 1131 256637 #> 2 1344 338267 #> 3 1746 390884 #> 4 2416 518459

  

  





That single question ("missed by less than how much?") is the whole method. The rest of the lesson is just answering it carefully, in four steps, in R.