Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 5 of 6

Monitoring and drift

Dev's meal-kit cancellation model launched three months ago at about 79% accuracy, and everyone moved on. This morning the retention team notices something odd: the discount emails the model targets are landing on subscribers who were never going to leave, while the ones quietly cancelling get nothing. Nobody changed a line of code. The model did not break. The world it was trained on did.

That is drift, and it is the quietest failure in machine learning: no error, no crash, just decisions getting worse while every dashboard stays green. This lesson is about catching it before your users do.

By the end of this lesson you will be able to:

  • Explain why a model that was accurate at launch decays even though its code never changes
  • Detect data drift by comparing live inputs to what the model trained on, and put a number on it with the population stability index (PSI)
  • Tell data drift from concept drift, and decide when a monitoring signal means it is time to retrain

Prerequisites: Lesson 4 (predictions flowing in batch or real time), and you can [fit a model and read predict output](Your-First-End-to-End-Model-in-R.html).

Why models decay

A model is a snapshot of a world that keeps moving

When Dev trained the cancellation model, it learned the patterns in one specific slice of time: last quarter's subscribers, last quarter's prices, last quarter's reasons for leaving. Training freezes those patterns into fixed coefficients. From that moment the model is a photograph, and photographs do not update themselves.

Then the world moves. Two months after launch a cheaper rival, FreshBox, opens for business. Boxes get discounted to compete, so the typical price a subscriber pays drifts down. And the reason people cancel changes: where idle weeks used to be the main signal, now even loyal, long-tenure customers leave, lured by the cheaper option. The model never saw any of this. It keeps applying last quarter's rules to this quarter's world, and quietly gets them wrong.

We do not have to take this on faith. Let us rebuild Dev's model from scratch, then score it on a fresh batch from the launch-era world and on a batch from the world three months later, and read the two accuracies side by side.

RInteractive R
# Dev's meal-kit cancellation model, rebuilt from scratch. Each lesson runs in a # fresh R session, so we simulate the subscribers it trained on, inline. clip <- function(x, lo, hi) pmax(lo, pmin(hi, x)) set.seed(1) n <- 1500 launch <- data.frame( weeks_since = rpois(n, 3), # weeks since last order boxes_ordered = rpois(n, 8) + 1, # lifetime boxes spend_per_box = round(clip(rnorm(n, 15, 3.5), 6, 24), 1) # dollars per box ) # The world at launch: idle weeks drive cancellation; price does not matter yet. launch$cancelled <- rbinom(n, 1, plogis(-3.5 + 1.15 * launch$weeks_since - 0.06 * launch$boxes_ordered)) model <- glm(cancelled ~ weeks_since + boxes_ordered + spend_per_box, data = launch, family = binomial) accuracy <- function(batch) mean((predict(model, batch, type = "response") > 0.5) == batch$cancelled) # A fresh batch from the world the model launched in set.seed(11) launch_batch <- data.frame( weeks_since = rpois(500, 3), boxes_ordered = rpois(500, 8) + 1, spend_per_box = round(clip(rnorm(500, 15, 3.5), 6, 24), 1)) launch_batch$cancelled <- rbinom(500, 1, plogis(-3.5 + 1.15 * launch_batch$weeks_since - 0.06 * launch_batch$boxes_ordered)) # Three months later: FreshBox (cheaper) has launched. Boxes are cheaper (spend drifts # down) and even loyal customers now leave, so price starts to predict churn. set.seed(12) later_batch <- data.frame( weeks_since = rpois(500, 3), boxes_ordered = rpois(500, 8) + 1, spend_per_box = round(clip(rnorm(500, 12.5, 3.5), 6, 24), 1)) later_batch$cancelled <- rbinom(500, 1, plogis(-2.55 + 1.15 * later_batch$weeks_since - 0.06 * later_batch$boxes_ordered + 0.07 * later_batch$spend_per_box)) round(c(launch = accuracy(launch_batch), later = accuracy(later_batch)), 2) #> launch later #> 0.79 0.65

  

  





Same model, same code, fourteen points of accuracy gone. Nothing alerted, because from the code's point of view nothing happened. The only way to catch this is to watch for it on purpose. The rest of the lesson is how.