Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 5 of 6

Calibrating Predicted Probabilities

In Lesson 4 you read the curves that judge a model across every threshold. Every one of them quietly trusted the score itself: that a transaction the model rates 0.8 really is fraud about 80% of the time. This lesson checks that assumption, and repairs it when it fails.

Think of a weather forecaster. When she says "70% chance of rain," you want it to actually rain on roughly 70 of every 100 such days. A model that says "0.7 chance of fraud" deserves the same test. When it passes, we call it calibrated, and only then can a probability be read at face value.

By the end you will be able to:

  • Explain what a calibrated probability is, and why a great AUC does not guarantee one
  • Read a reliability diagram and tell an over-confident model from an under-confident one
  • Put a single number on calibration with the Brier score and expected calibration error
  • Repair a miscalibrated model in R with Platt scaling and isotonic regression, the leak-free way

Prerequisites: you can fit a classifier in R and know a train/test split (Train, Validation, Test and Data Leakage), and you know a classifier outputs a probability score (Lesson 3). In Lesson 2 you oversampled the rare class to lift recall; this lesson cleans up the probabilities that trick left behind.

What it means

What a calibrated probability means

A calibrated probability is one you can take at face value. Formally, a model is calibrated when, among all the transactions it scores at some value \(p\), the fraction that turn out to be fraud is exactly \(p\):

\[ P(Y = 1 \mid \hat p = p) = p \qquad \text{for every } p. \]

Here \(\hat p\) is the model's predicted probability of fraud and \(Y\) is the true outcome (1 for fraud, 0 for legit). In words: gather every transaction the model rated 0.30; if the model is calibrated, about 30% of them really were fraud. That is exactly how you would grade the weather forecaster, and exactly the standard a risk score must meet before anyone acts on the number.

Calibration is a different job from ranking. Lesson 4's AUC only asked whether fraud tends to outscore legit; it never asked what the scores actually were. So a model can rank beautifully and still be badly miscalibrated. To watch that happen, we rebuild the fraud detector, oversampled exactly the way Lesson 2 did it.

RInteractive R
# The bank's fraud detector, rebuilt from scratch (each lesson is a fresh R session). # Fraud is rare and spends a little differently. Build the three sets we will need. set.seed(42) make_txns <- function(n_legit, n_fraud) { legit <- data.frame(amount = round(rlnorm(n_legit, 3.3, 0.8), 2), foreign = rbinom(n_legit, 1, 0.04), class = 0) fraud <- data.frame(amount = round(rlnorm(n_fraud, 3.9, 0.9), 2), foreign = rbinom(n_fraud, 1, 0.30), class = 1) d <- rbind(legit, fraud) d[sample(nrow(d)), ] # shuffle the rows } train <- make_txns(1900, 100) # 5% fraud - the model is fit on this calib <- make_txns(950, 50) # 5% fraud - HELD OUT, to learn the calibration fix later test <- make_txns(950, 50) # 5% fraud - untouched, for the honest final check table(train$class) #> #> 0 1 #> 1900 100

  

  





Now the miscalibration. In Lesson 2 you oversampled the rare fraud rows to lift recall, and I warned it distorts the probabilities. Here is that distortion in the open: balance the training rows 50/50, fit a plain logistic regression, and ask what it predicts.




  

    
RInteractive R

    
# Balance the TRAINING rows 50/50 by duplicating fraud (Lesson 2's oversampling), then fit.
set.seed(1)
min_i     <- which(train$class == 1)
extra     <- sample(min_i, sum(train$class == 0) - length(min_i), replace = TRUE)
train_bal <- train[c(seq_len(nrow(train)), extra), ]

fit    <- glm(class ~ amount + foreign, data = train_bal, family = binomial)
p_cal  <- predict(fit, calib, type = "response")   # scores on the held-out calibration set
p_test <- predict(fit, test,  type = "response")   # scores on the untouched test set

# The model trained as if fraud were common, so on average it "sees" it everywhere.
round(c(mean_predicted = mean(p_test), actual_fraud_rate = mean(test$class)), 3)
#> mean_predicted actual_fraud_rate 
#>          0.416             0.050

    

      
      
    

    

  

  





The model predicts fraud with an average probability of 0.42, on a set where only 5% of transactions really are fraud, more than eight times too high. That is over-confidence, and next we draw it.