Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 5 of 7

Calibration, Reliability and Recalibration

For four lessons you have been building things that report a number and asking it to be honest: a prediction interval that really covers 90% of the time, a quantile band whose width breathes with the data. Every one of those promises rested on a probability meaning what it says. This lesson finally puts that assumption on trial.

Meet Nadia. She built the rain model behind a bike-courier app: every morning it gives each neighborhood a probability of rain, and the dispatcher pays for extra rain-gear couriers wherever that number tops 0.6. So the number is not decoration, it spends real money. If the model says "0.90 chance of rain" on mornings when it actually rains only about eight times in ten, Nadia is over-provisioning gear she did not need. Her model is good at ranking the wet mornings above the dry ones, and yet its probabilities can still be lies. Telling those two things apart, and repairing the second without touching the first, is the whole lesson.

By the end you will be able to:

  • Say exactly what a calibrated probability is, and read a reliability diagram to spot an over-confident or under-confident model
  • Put a single number on miscalibration with the expected calibration error and the Brier score, in R
  • Recalibrate a model with Platt scaling and isotonic regression, on a held-out set, and prove the fix worked while its ranking power stayed identical

Prerequisites: Lessons 1 to 4 of this course (prediction intervals, coverage, and the held-out-split idea you met in conformal). Base R: glm(), plogis()/qlogis(), predict(), logical indexing, and writing a small function. Every new term is defined as it appears.

Where we left off

A predicted probability is a promise

In Lesson 4 you priced Rohan's homes, a number outcome, and asked whether an interval covered it. Nadia's world is different in one way that changes everything: her outcome is a yes/no event (did it rain?), and her model reports a probability. There is no interval to check. Instead there is a subtler promise to keep.

A probability is calibrated when it matches long-run reality. Take every morning where the model said "0.70 chance of rain." If it truly rained on about 70% of those mornings, the 0.70 was honest. If it rained on only 55% of them, the model was over-confident: it said 0.70 but the world delivered 0.55. Calibration is that simple, and that demanding, promise: of all the times you say q, a fraction q should come true.

Let us build Nadia's world so we can test it. Each lesson runs in a fresh R session, so we create the data right here (run this once). We simulate an honest rain chance from the morning's humidity, draw whether it actually rained, and then, crucially, give the model a score that ranks mornings correctly but is too extreme, the single most common way real models (boosted trees, neural nets, naive Bayes) go wrong.

RInteractive R
set.seed(11) n <- 4000 humidity <- runif(n, 0, 1) # 0 = bone dry air, 1 = saturated true_lo <- -0.3 + 4.6 * (humidity - 0.5) # the honest log-odds of rain true_p <- plogis(true_lo) # the honest probability rain <- rbinom(n, 1, true_p) # did it actually rain? 1 = yes # Nadia's model: SAME ranking as the truth, but over-confident (log-odds stretched 1.6x) model_p <- plogis(1.6 * true_lo) dat <- data.frame(humidity, model_p, rain) cal <- dat[1:2000, ] # a held-out CALIBRATION set (we will need it later) test <- dat[2001:4000, ] # a separate set for honest scoring round(range(model_p), 3) #> [1] 0.015 0.961 c(base_rate = round(mean(test$rain), 3)) #> base_rate #> 0.456

  

  





The model issues the full range of confidences, from 0.015 to 0.961, and it rains on about 46% of test mornings overall. Stretching the log-odds by 1.6 keeps the order of the mornings exactly as the truth had it (a bigger honest chance is still a bigger score), so the model ranks perfectly well. What it breaks is the values: high scores are pushed too high and low scores too low. Now we need a way to see that.