Adapting to Drift: Reweighting and Retraining

In Lesson 2, Nadia's monitor finally earned its keep. With no fraud labels in hand, the Population Stability Index caught the purchase amount creeping up and tripped the alarm. But an alarm is not a fix. Knowing the inputs moved tells you nothing about what to do next.

There are exactly two moves, and this lesson is about choosing between them and executing each one in R:

• Reweight the data you already have, no new labels needed, when only the inputs drifted.
• Retrain on fresh labels, when the world's actual rule changed, and do it on a trigger, not a calendar.

Slide the monitor below: as the weeks pass, the live inputs pull away from what Nadia's model trained on and the PSI climbs past its alarm line. That moment, when the alarm fires, is where this lesson begins.

By the end you will be able to:

• Decide which fix a shift needs: covariate shift you reweight, concept shift you retrain
• Correct covariate shift by importance-weighting the training data with a density ratio
• Estimate those weights the easy way, as the odds of a domain classifier (the Lesson 2 trick, reused)
• Set a control-limit retraining trigger so you retrain when the data demands it, not on a schedule

Prerequisites: you finished Lesson 2 (PSI and its 0.1 / 0.2 lines, the classifier two-sample test, covariate vs concept shift), and you can fit and read a logistic regression.

Two shifts, two fixes

Lesson 1 split the ways a deployed model can break. Two of them matter here, and they call for opposite responses.

Covariate shift means the inputs moved but the rule did not: \(P(x)\), how common each feature value is, changes, while \(P(y \mid x)\), the chance of fraud given those features, stays fixed. For Nadia, customers started making bigger purchases, so the distribution of amount shifted up, but a transaction of a given size is still exactly as likely to be fraud as it always was. The map from features to fraud is intact; you just have fewer training examples in the region where traffic now lives. You do not need new labels to fix this. You reweight.

Concept shift means the rule itself moved: \(P(y \mid x)\) changed. Fraudsters found a new trick, so transactions that used to look safe are now fraud. No reweighting of old data can teach the new rule, because that old data was labelled under the old rule. You need fresh labels and a retrain.

We take them in that order: reweighting first, then when and how to retrain.