From Metrics to Money
In Lesson 6 you learned to tell a real metric difference between two models from the ordinary luck of a single split. So now you can say, with a straight face, that your new model is genuinely better.
Here is where that runs into a wall. Maya runs customer retention at FreshBox, a meal-kit company. Her data scientist just shipped a churn model with an AUC of 0.83, a real, tested improvement over the old one. Maya's boss asks a simpler question: "Good. How many dollars a month is that worth?" The AUC is not an answer to that question. Nothing on the metrics dashboard is.
This lesson builds the bridge from a score to a number in dollars.
By the end you will be able to:
- Put a dollar value on each cell of a confusion matrix, and write a model's profit as one formula
- Set the decision threshold from the costs, and see why 0.5 is almost always the wrong line to draw
- Say what a metric gain is actually worth, and why a bigger AUC is not automatically more money
Prerequisites: you can run R and write a small function, and from Lesson 5 you have met the confusion matrix, the ROC curve and AUC.
A prediction is only worth what it changes
A churn score sitting in a database earns nothing. It becomes worth something only when it triggers an action. At FreshBox that action is concrete: for every customer the model flags as likely to churn, Maya's team sends an $8 retention offer (a discount coupon). If the customer really was about to leave and the offer wins them back, FreshBox keeps a customer worth $120 in future profit. If the customer was never going to leave, the $8 is simply wasted.
So each customer falls into one of four outcomes, the same four cells of the confusion matrix you already know, but now each cell is worth a different amount of money. Slide the threshold on the classifier below and watch those four counts move; that is the machine we are about to price.
Here is the price tag on each cell, measured against the alternative of running no offers at all:
| Outcome | What happens | Dollar value |
|---|---|---|
| Save (true positive) | offer a real churner, win them back | +$112 |
| Waste (false positive) | offer a customer who would have stayed | -$8 |
| Miss (false negative) | a real churner you never contacted | $0 |
| Correct pass (true negative) | a loyal customer you left alone | $0 |
The two zeros surprise people, so let us be exact about them. We measure every cell against doing nothing. If we run no offers, the churner leaves anyway, so failing to contact one (a miss) changes nothing versus that baseline: $0. Likewise, correctly leaving a loyal customer alone costs and earns nothing: $0. The program only ever makes or loses money in the top row, when it acts: a save earns \(+\$112\) (the \$120 kept minus the \$8 offer), a wasted offer costs \(-\$8\).
That gives the whole program's profit as one clean formula. With \(b = 112\) the benefit of a save and \(c = 8\) the cost of a wasted offer:
\[ \Pi = b \cdot \mathrm{TP} - c \cdot \mathrm{FP} \]
where \(\mathrm{TP}\) (true positives) is the number of real churners you contacted and \(\mathrm{FP}\) (false positives) is the number of stayers you contacted. Every dollar the model is worth lives in that single line.
Let us set up FreshBox's month and price today's policy. Each lesson runs in a fresh R session, so we build everything inline (run this once):
Read that last line and sit with it. Today's perfectly reasonable-sounding rule, "act on customers who are more likely than not to churn," earns FreshBox exactly $0, because in a given month no customer is more than about 30% likely to cancel, so the offer never goes out. A model that is never acted on is worth nothing, no matter how good its AUC.