Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 1 of 7

Kinds of Distribution Shift

Nadia builds a fraud detector for a payments company. It learns from a year of labelled card transactions and, at launch, catches fraud well enough to ship. Six months later it is missing a wave of fraud it should have caught, yet not one line of code changed and no server ever threw an error. What changed was the world the model lives in.

When the data a model meets in production drifts away from the data it trained on, we call it distribution shift. This lesson names the three kinds, and shows which one is merely a nuisance and which one silently makes a good model wrong, all on Nadia's detector.

By the end of this lesson you will be able to:

  • Name the three kinds of distribution shift by which part of the data changed
  • Explain why some shifts leave a good model working while one kind breaks it
  • Diagnose, from a described change in the real world, which shift you are facing

Prerequisites: you can fit and read a classifier (such as a logistic regression), and you know what a decision threshold and a held-out set are.

The setup

Train here, deploy there

A model only ever sees its training data. Nadia's detector learned from transactions each labelled fraud or legit over the past year. Write one transaction as a pair: its features \(x\) (here, the purchase amount) and its label \(y\) (1 for fraud, 0 for legit). Together they are drawn from a joint distribution \(P(x, y)\), the full pattern of what transactions look like and how often each one is fraud.

That joint splits into three pieces we will need by name:

  • \(P(x)\): how the features are spread out, how big purchases tend to be.
  • \(P(y)\): the base rate, the fraction of all transactions that are fraud.
  • \(P(y \mid x)\): the rule linking them, "given a purchase of this size, how likely is fraud." This is exactly what a classifier learns.

Let us build Nadia's world and fit her model, so we have something concrete to break. Each lesson runs in a fresh R session, so we create the data right here.

RInteractive R
set.seed(1) n <- 4000 amount <- rnorm(n) # purchase size, standardized (a z-score) fraud <- rbinom(n, 1, plogis(1.5 * amount)) # launch rule: bigger amounts, more fraud launch <- data.frame(amount, fraud) model <- glm(fraud ~ amount, data = launch, family = binomial) # accuracy: the share of transactions the 0.5 cut-off labels correctly acc <- function(m, d) mean((predict(m, d, type = "response") > 0.5) == d$fraud) round(acc(model, launch), 3) #> [1] 0.737

  

  





The model labels about 74% of transactions correctly, good enough to ship. Hold on to model and acc; we will point them at three different futures and see what happens.