Thresholds Under Asymmetric Costs

SecureBank's fraud model looks at a $900 online purchase and returns one number: a 34% chance it is fraud. The model has done its job. Now a human decision is left: flag it, or let it through? Answer "flag it only if the chance is above 50%" and the bank waves through every fraud the model rated under a coin flip. Each one it waves through is a $500 chargeback.

In Lesson 2 you rebalanced the training data to catch more fraud, and paid for it in precision and distorted probabilities. This lesson does not touch the data or retrain anything. It moves one dial, the decision threshold, and lets the cost of each mistake decide where that dial should sit.

By the end you will be able to:

• Explain why the default 0.5 cutoff is arbitrary, and that moving it changes predictions with no retraining
• Read an asymmetric cost matrix and decide which way to move the threshold
• Price a set of predictions in R and sweep the threshold to find the cheapest cutoff
• Use the cost-optimal threshold formula to pick a cutoff directly

Prerequisites: you can fit a classifier that outputs a probability (Logistic Regression Done Properly), and you know precision, recall and a confusion matrix (Class Imbalance and Resampling).

Drag the threshold below and watch the flagged points, and the mistakes, change.

The score, not the label

A classifier does not really output "fraud" or "legit". It outputs a probability: SecureBank's model gives that $900 purchase a fraud probability of \(p = 0.34\). The yes/no label only appears when you compare that probability to a cutoff.

That is the decision rule, and it is the whole game:

\[ \hat{y} = 1 \iff p \ge t \]

Read that as: predict fraud exactly when the probability is at least the cutoff. Here \(p\) is the model's predicted probability of the positive class (fraud), \(t\) is the threshold you choose, \(\hat{y} = 1\) means "predict fraud" (the little hat marks it as the model's guess, not the truth), and the \(\iff\) symbol means "exactly when". Most software defaults to \(t = 0.5\), so people forget it is a choice at all. It is not a law of the model; it is a knob. Slide the threshold on the curve in the previous step and the same fixed S-curve keeps its shape while the cutoff line, and the set of flagged transactions, moves.