Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 7 of 7

A Monitoring and Robustness Playbook

Six lessons, six ways a deployed model quietly fails: the inputs drift, a strange request arrives, a subgroup is sacrificed to the average, an adversary aims at the boundary. You now know how to detect each one. This final lesson turns that pile of detectors into a single thing you can actually run in production: a playbook.

Meet Nadia one last time. Her fraud model, fraud-v3, is live, scoring every transaction the moment it happens. But "live" is not "done." The true fraud labels for today's transactions will not arrive for weeks, an adversary is probing the filter right now, and a new customer segment is behaving in ways the model never saw in training. Nadia needs a plan for what to watch, and, when something trips, a rule for who or what responds. The monitor below is where that plan begins: slide it forward and watch a single number climb until it trips an alarm.

By the end of this lesson you will be able to:

  • Specify what a live model must log so every downstream check is even possible
  • Assemble the section's detectors into a layered alarm stack, and know each layer's cadence and blind spot
  • Write a severity-to-response policy, and state precisely when an alarm should page a human versus roll back automatically
  • Run a pre-deploy gate that pairs a worst-group accuracy floor with an adversarial evasion ceiling before any model ships

Prerequisites: you finished Lessons 1 to 6 of this course, covering drift kinds, detecting drift (PSI), reweighting and retraining triggers, out-of-distribution detection, group robustness, and adversarial robustness. You can fit and read a logistic regression and read a control chart.

The foundation

What to log

Every check in this playbook reads from one place: the prediction log. If a field is not logged when a transaction is scored, it is gone, and no monitor can ever recover it. So the playbook starts before any alarm, with the humble question of what to write down for each scored request.

Four things, and each earns its place:

  1. The inputs the model actually saw (amount, mrisk). Drift and out-of-distribution checks compare these against training.
  2. The prediction and its score (pred, score). The 0-to-1 score, not just the yes/no call, is what lets you watch confidence and calibration.
  3. The model version and batch (model, batch). When you run two models or roll back, you must be able to say which model made a call and when.
  4. A slot for the label (label), left empty now. The ground truth, was this transaction really fraud, arrives weeks later and is joined back on. Every performance metric waits on this join.

Each lesson runs in its own R session, so we build Nadia's world from scratch: her training data, the deployed fraud-v3 model fit on it, and one live week of scored, logged transactions. Notice the new-market segment sends larger baskets, and today's labels are all NA.

RInteractive R
set.seed(11) rule <- function(a, m) plogis(-1.6 + 1.4 * a + 1.3 * m) # true fraud risk, fixed ntr <- 5000 train <- data.frame(amount = rnorm(ntr), mrisk = rnorm(ntr)) # standardized features train$fraud <- rbinom(ntr, 1, rule(train$amount, train$mrisk)) model <- glm(fraud ~ amount + mrisk, family = binomial, data = train) # deployed fraud-v3 set.seed(40) n <- 4000 region <- sample(c("core", "new_market"), n, TRUE, c(0.8, 0.2)) amount <- rnorm(n, mean = ifelse(region == "new_market", 1.7, 0.6)) # this week's live inputs mrisk <- rnorm(n, mean = 0.1) live <- data.frame(amount, mrisk, region) live$score <- predict(model, live, type = "response") log_week <- data.frame( batch = "2026-W40", model = "fraud-v3", amount = round(live$amount, 2), mrisk = round(live$mrisk, 2), region = live$region, pred = as.integer(live$score > 0.5), score = round(live$score, 3), label = NA_integer_) # true outcome confirmed weeks later head(log_week, 3) #> batch model amount mrisk region pred score label #> 1 2026-W40 fraud-v3 1.95 -0.77 core 1 0.543 NA #> 2 2026-W40 fraud-v3 1.95 1.46 new_market 1 0.952 NA #> 3 2026-W40 fraud-v3 1.17 0.64 core 1 0.701 NA

  

  





Key Insight
Logging is not paperwork, it is the whole game. A field you did not log is a check you cannot run. Log the inputs, the score, the model version, and reserve the slot for the late label, and every alarm below becomes possible.