Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 4 of 7

Out-of-Distribution and Novelty Detection

In Lesson 3, Nadia's monitor did its job: the Population Stability Index caught the whole stream of transactions drifting, and you learned the two fixes, reweight or retrain. But drift is a question about the crowd. It asks whether the population as a whole has moved.

This lesson asks a sharper, lonelier question about a single transaction: is this one unlike anything the model ever trained on? A drift monitor would happily wave through one bizarre payment hiding among a thousand ordinary ones, because on average the batch still looks fine. Novelty detection is the per-input alarm that catches the individual oddity.

The widget below shows the score you are about to build. The pile on the left is ordinary transactions; the pile on the right is genuinely strange ones. Toggle the cutoff between loose, medium and strict and watch the two numbers move: every novelty detector lives on that one dial, and by the end of this lesson you will set it on purpose.

By the end of this lesson you will be able to:

  • Explain why a trained model returns a confident answer even on an input unlike anything it saw, and why that is dangerous
  • Score how unusual a single input is with the Mahalanobis distance, and say why it beats a per-feature or straight-line (Euclidean) check
  • Turn that score into a calibrated flag with a chi-square threshold, and read the false-positive versus detection-rate tradeoff the threshold sets
  • Name where a single-Gaussian novelty score breaks, and what to reach for instead

Prerequisites: you finished Lesson 3: Adapting to Drift (drift is a stream-level question; PSI and the two-sample tests look at one feature at a time), you can fit and read a logistic regression, and you know what a covariance and a correlation are.

Why we need a second opinion

The model that cannot say "I have never seen this"

Here is the uncomfortable truth about Nadia's fraud model, and about almost every model in production: it will answer any question you ask it, even a nonsensical one. Hand a logistic regression a transaction from a planet it has never visited and it will still compute a tidy probability and hand it back with a straight face. It has no way to say "this input is unlike my training data, so do not trust me here."

Let us watch it happen. We rebuild Nadia's world from scratch (each lesson runs in its own R session). Every transaction has two features: account_age, the days since the customer signed up, and amount, the size of the order in dollars. In her real customer base these two move together, because older accounts tend to place bigger orders, so the features carry a positive correlation of about 0.79.

RInteractive R
set.seed(1) n <- 2000 account_age <- pmax(round(rnorm(n, 500, 180)), 5) # days since sign-up amount <- round(pmax(50 + 0.42 * account_age + rnorm(n, 0, 60), 1), 1) # order size, dollars train <- data.frame(account_age = account_age, amount = amount, fraud = rbinom(n, 1, plogis(-5.2 + 0.006 * amount))) model <- glm(fraud ~ account_age + amount, data = train, family = binomial) young_big <- data.frame(account_age = 90, amount = 360) # a brand-new account, a large order round(unname(predict(model, young_big, type = "response")), 3) #> [1] 0.065

  

  





The model reads that transaction, a 90-day-old account placing a 360-dollar order, and calmly returns a fraud probability of 0.065. Approve it. No hesitation, no warning.



But look at what that transaction actually is. Nadia's accounts range from days old to over three years (age runs 5 to 1186 days), and orders routinely top 600 dollars, so taken one at a time neither number is strange. What is strange is the pairing: in the training data a barely-90-day-old account almost never places a 360-dollar order, because young accounts and big orders rarely go together. The model has essentially no examples like this, so its 0.065 is not a measurement, it is an extrapolation dressed up as one.



Key Insight
A trained classifier interpolates confidently and extrapolates confidently, and from the outside the two look identical. It cannot flag its own blind spots. Novelty detection is a separate model whose only job is to answer "have we seen anything like this before?"