Reading a Classifier

In Lesson 5 you learned to see a classifier as the boundary it draws. But a boundary that looks tidy is not the same as a boundary that works, and to tell them apart you need numbers. This lesson is the scorecard: how to grade a trained classifier honestly, and why the most obvious number, accuracy, can hide a nearly useless model.

Meet our running example. The Riverbend Clinic runs a quick blood-marker model to screen walk-in patients for a thyroid condition that only about 1 patient in 8 actually has. When Dana walks in, the model reads her marker and returns a single number: a risk score of 0.61. Is that a "yes"? Only once we pick a cutoff. And once the clinic starts flagging patients, every prediction lands in one of four buckets, and those four numbers decide whether the screen is trustworthy or dangerous.

By the end of this lesson you will be able to:

• Read a confusion matrix and name which of its four cells hurts most in a given problem
• Compute and interpret accuracy, precision, recall, specificity and F1, and explain the accuracy paradox
• Explain how a threshold turns a score into a label, and how moving it trades precision against recall
• Read an ROC curve and its AUC, and a precision-recall curve, and know when each tells the honest story
• Choose a threshold from the real cost of a false alarm versus a missed case

Prerequisites: you can run R and read its output, and you have met logistic regression giving a probability \(P(\text{class} \mid x)\) and the idea of a decision boundary (Lesson 5).

The panel above is the whole scorecard in miniature: a confusion matrix, an ROC curve and one AUC number, all reacting to a single threshold dial (on illustrative scores for now). By the end of this lesson every part of it will make sense. Slide it if you like; then let us build it up from scratch.

A classifier hands you a score, not a verdict

Here is the idea that everything else hangs on. A trained classifier like the Riverbend model does not really output "sick" or "healthy". It outputs a score: a number, usually a probability between 0 and 1, saying how confident it is. Dana's score was 0.61. To turn that score into an actual decision, you must pick a threshold \(t\) and apply a simple rule:

\[ \text{predict positive} \quad\text{when}\quad \hat{p}(x) \ge t \]

Here \(\hat{p}(x)\) is the model's estimated probability that patient \(x\) is sick, and \(t\) is the cutoff you choose. The default is \(t = 0.5\): flag anyone the model thinks is more likely sick than not. With that cutoff Dana (0.61) is flagged; a patient scoring 0.30 is cleared.

The crucial part: the threshold is a knob, not a fact. The model's scores are fixed once it is trained, but you can slide the cutoff anywhere from 0 to 1, and every choice produces a different set of predictions, and therefore a different scorecard. The widget below shows exactly that. Each dot is a patient positioned by score; slide the vertical cutoff and watch dots cross from one predicted label to the other.

Move the cutoff left and you flag more patients: you catch more of the truly sick, but you also raise more false alarms. Move it right and the opposite happens. That single trade is the engine of this entire lesson. First, though, we need to count the outcomes.