Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 4 of 6

ROC, PR, Lift and Gains Curves

In Lesson 3 you stood at one threshold and priced its mistakes. But one threshold is one snapshot. This lesson steps back and looks at the model across every threshold at once, through four famous curves: the ROC curve, the precision-recall curve, the cumulative gains curve, and the lift curve.

Here is the running story. Overnight, SecureBank's model scored a full day of 3000 card transactions for fraud. Only about 1 in 20 is actually fraud (the rest are ordinary purchases), and the bank's fraud team can hand-review only a slice of the day. Four questions hang over that morning: does the model rank fraud above legit? When it raises an alarm, can you trust it? If you review the riskiest 10%, how much fraud do you catch? And how much better is that than reviewing at random? Each of the four curves answers exactly one of them.

By the end of this lesson you will be able to:

  • See ROC, PR, gains and lift as four questions asked of one score-ranked list
  • Read an ROC curve and its AUC, and explain why a great AUC can hide a poor model when positives are rare
  • Read a precision-recall curve against the right baseline, and summarize it with AUPRC
  • Read cumulative gains and lift curves to decide how far down a ranked list to act

Prerequisites: a classifier outputs a probability score and a threshold turns it into a label (Thresholds Under Asymmetric Costs), and you have met the confusion matrix, precision, recall and AUC at an intro level (Reading a Classifier). We re-define every term as it appears.

The organizing idea

One ranked list, four questions

Strip away the jargon and a scored classifier gives you just one thing: a ranked list. Sort SecureBank's 3000 transactions from the highest fraud score to the lowest, and every curve in this lesson is a different question asked of that one list. ROC, PR, gains and lift do not need four different models; they re-read the same ranking four ways.

So let us build that list. Each lesson runs in a fresh R session, so we create the day of scored transactions right here (run this once). We simulate 3000 transactions, mark each as fraud or legit (fraud is rare, about 5%), and give each a model score between 0 and 1 that runs higher for fraud but overlaps heavily with legit, exactly what a good-but-imperfect model produces.

RInteractive R
set.seed(2024) n <- 3000 # 1 = fraud (the rare positive, about 5%), 0 = an ordinary legit purchase. fraud <- rbinom(n, 1, 0.05) # The model's fraud score: higher on fraud, but the two clouds overlap heavily. score <- plogis(rnorm(n, mean = -2.7 + 2.6 * fraud, sd = 1.2)) txn <- data.frame(fraud = fraud, score = round(score, 4)) table(fraud = txn$fraud) #> fraud #> 0 1 #> 2852 148

  

  





Of the 3000 transactions, 148 are truly fraud and 2852 are legit. That fraction of positives has a name we will use constantly, the prevalence (also called the base rate): the share of all cases that are truly positive.




  

    
RInteractive R

    
prevalence <- mean(txn$fraud)
round(prevalence, 3)
#> [1] 0.049

    

      
      
    

    

  

  





About 0.049, just under 5%. Hold onto that number. On a rare-positive problem it is the single fact that decides which curve tells the truth.