Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 2 of 6

Class Imbalance and Resampling

Priya is a fraud analyst at a bank. Last month's data has 10,000 card transactions, and exactly 52 of them were fraud. That is one fraud for every 191 legitimate transactions. She trains a classifier, and it comes back 99.5% accurate. She is thrilled, until she checks how many of the 52 frauds it actually caught: zero.

That is the trap of an imbalanced problem, where one class is far rarer than the other. This lesson is about seeing the trap, and the main tool for climbing out of it: resampling the training data so the rare class stops being invisible.

By the end you will be able to:

  • Explain why plain accuracy is misleading when one class is rare, and read recall and precision instead
  • Tell undersampling, oversampling and SMOTE apart, and describe what each does to the training rows
  • Rebalance a training set in R and measure whether it actually helped
  • Resample the training data only, so you never leak the fix into your own evaluation
  • Judge when to reach for resampling, and when a different tool is better

Prerequisites: Lesson 1 of this course, Beyond Binary: Multiclass Classification (the confusion matrix, precision and recall); a classifier that outputs a probability, from Reading a Classifier; a train/test split and what leakage is; and you can fit a model in R.

The panel below is Priya's problem in miniature: a sea of ordinary transactions (blue) with a handful of frauds (orange) huddled in one corner. Toggle the buttons to see the two fixes this lesson builds.

The trap

The 99.5% trap

Let us rebuild Priya's month from scratch so every number here is real. Each transaction has a size, an hour, a count of recent declined attempts, and whether the card is foreign. Fraud is rare and tends to be large, late, foreign, and preceded by declines.

RInteractive R
set.seed(1) n <- 10000 tx <- data.frame( amount = round(rlnorm(n, 3.4, 1.0), 2), # transaction size in dollars hour = sample(0:23, n, replace = TRUE), # hour of day, 0 to 23 n_declines = rpois(n, 0.3), # recent declined attempts foreign = rbinom(n, 1, 0.08) # foreign card? 1 = yes ) risk <- -6.4 + 0.004*tx$amount + 0.8*(tx$hour < 5) + 0.9*tx$n_declines + 1.1*tx$foreign tx$fraud <- factor(ifelse(runif(n) < plogis(risk), "yes", "no"), levels = c("no", "yes")) table(tx$fraud) #> #> no yes #> 9948 52

  

  





There it is: 9,948 legitimate transactions and 52 frauds. Now split off a test set to grade the model honestly, exactly as you would in real life.




  

    
RInteractive R

    
set.seed(7)
train_idx <- sample(nrow(tx), 7000)   # 70% to train on
train <- tx[train_idx, ]
test  <- tx[-train_idx, ]             # 30% held out, never trained on
table(test$fraud)
#>
#>   no  yes
#> 2986   14

    

      
      
    

    

  

  





Fourteen of the test frauds are what we need the model to catch. Fit a logistic regression on the training data and look at how it does on the held-out test set. The table has the truth down the rows and the model's guess across the columns.




  

    
RInteractive R

    
model <- glm(fraud ~ amount + hour + n_declines + foreign, data = train, family = binomial)
pred  <- factor(ifelse(predict(model, test, type = "response") > 0.5, "yes", "no"),
                levels = c("no", "yes"))
table(actual = test$fraud, predicted = pred)
#>       predicted
#> actual   no  yes
#>    no  2986    0
#>    yes   14    0
mean(pred == test$fraud)   # overall accuracy
#> [1] 0.9953333

    

      
      
    

    

  

  





Look at that bottom row. The model labelled every single transaction "no fraud". It caught 0 of the 14 frauds, and still scored 99.5% accuracy, because guessing the majority class is almost always right when the majority is 99.5% of the data. Accuracy rewarded the model for ignoring exactly the thing Priya cares about.



Key Insight
When one class is rare, a model can score sky-high accuracy by never predicting the rare class at all. Accuracy measures the wrong thing here. You need a score that notices the minority.