Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 2 of 7

Detecting Distribution Shift

In Lesson 1, Nadia's fraud detector quietly broke when fraudsters changed tactics. The damage was real, but its accuracy only revealed it weeks later, once the chargebacks came in and the true fraud labels finally caught up. In production that lag is normal: labels are slow, expensive, or never arrive at all.

So you cannot sit and watch accuracy. You have to notice the world moving from the inputs alone, the transaction features streaming in today, long before anyone confirms which were fraud. Slide the monitor below: as the weeks pass, this week's live feature drifts away from what the model trained on, and a single number climbs until it trips an alarm.

By the end of this lesson you will be able to:

  • Compute and read the Population Stability Index (PSI) on a feature, using the 0.1 and 0.2 thresholds
  • Run and interpret a Kolmogorov-Smirnov two-sample test comparing training data to live data
  • Build a classifier two-sample test, training a model to tell training from live and reading its AUC as a drift score
  • Pick the right detector and know each one's blind spot

Prerequisites: you finished Lesson 1, you can fit and read a classifier, and you know what a histogram, a p-value, and AUC are.

The problem

You cannot wait for the labels

Drift detection reframes the question. You are not asking "did the model get this transaction right?", because the answer arrives too late. You are asking a question you can answer today from features alone:

Does the data coming in now look like the data the model trained on?

That is a two-sample question. Hold a fixed reference sample, the feature values the model learned from, and a current window, a recent batch of live values. If those two samples plausibly came from the same distribution, you are stable. If they clearly did not, something moved.

Let us set up Nadia's monitor concretely. We watch one feature, the purchase amount, standardized so the average transaction sits at 0 and one unit means one standard deviation (that is why the numbers below hover around zero and can go negative, they are no longer raw dollars). We build the training reference, a later week that is genuinely unchanged, and a later week whose amounts have all crept up by about 0.6 standard deviations (bigger baskets: the covariate shift from Lesson 1, where the inputs move but the labelling rule does not). Each lesson runs in a fresh R session, so we create all three right here.

RInteractive R
set.seed(1) n <- 2000 ref <- rnorm(n, mean = 0, sd = 1) # what the model trained on stable <- rnorm(n, mean = 0, sd = 1) # a later week: inputs unchanged drifted <- rnorm(n, mean = 0.6, sd = 1) # a later week: baskets grew round(c(ref = mean(ref), stable = mean(stable), drifted = mean(drifted)), 3) #> ref stable drifted #> -0.014 0.016 0.584

  

  





We now have a truth to detect: stable really is drawn like ref, and drifted really is not. A good detector should stay quiet on stable and fire on drifted. We will run three of them, from the simplest to the most powerful.