Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 6 of 6

Why AUC Is Not Enough

The bank's data team ships two fraud detectors. Both score AUC 0.90 on the same test set. The slide deck says "great model" and picks one at random. In production, one detector saves the bank millions and the other floods the review team with false alarms and misprices every risk decision.

One number could not tell them apart. This lesson is about everything a single AUC hides, and the handful of numbers that would have caught the difference.

By the end you will be able to:

  • Say exactly what one AUC number measures, and what it does not
  • See why AUC is blind to whether a predicted 0.7 really means 70%
  • See why a high AUC can sit on top of terrible precision when the positive class is rare
  • Report the metric that matches the decision, instead of trusting AUC alone

Prerequisites: you can fit a classifier and read a confusion matrix. From earlier in this course: the ROC curve, precision and recall (ROC, PR, Lift and Gains Curves), reliability diagrams (Calibrating Predicted Probabilities), and why accuracy lies on a rare class (Class Imbalance and Resampling).

The idea

What one AUC number actually means

Start with what AUC is doing right, because it is a genuinely useful number. Take one real fraud transaction and one real legit transaction at random. Ask your model to score both. AUC is simply the probability that the fraud gets the higher score.

That is the whole definition. Write \(s(x)\) for the score a model gives a transaction \(x\). With \(x^{+}\) a random actual-fraud case and \(x^{-}\) a random actual-legit case,

\[ \text{AUC} = P\big(s(x^{+}) > s(x^{-})\big) \]

An AUC of 1.0 means every fraud outscores every legit (perfect ranking); 0.5 means the model orders them no better than a coin flip. It equals the area under the ROC curve you met in Lesson 4, and it is also the Mann-Whitney U statistic, so you can compute it by hand from scores and labels, no package required:

RInteractive R
# AUC = the chance a random positive is scored above a random negative. set.seed(1) n <- 1000 fraud <- rbinom(n, 1, 0.5) # a balanced toy set, to define AUC cleanly score <- plogis(rnorm(n, mean = ifelse(fraud == 1, 1, -1))) auc <- function(score, y) { # Mann-Whitney form, base R only pos <- score[y == 1]; neg <- score[y == 0] wins <- outer(pos, neg, ">") # every positive-vs-negative pair ties <- outer(pos, neg, "==") (sum(wins) + 0.5 * sum(ties)) / length(wins) # fraction of pairs ranked correctly } round(auc(score, fraud), 2) #> [1] 0.92

  

  





Key Insight
AUC measures ranking only: can the model put positives above negatives? It says nothing about the actual score values, nothing about the class balance, and nothing about the one threshold you will deploy at. Those three blind spots are the rest of this lesson.