Lesson 3 of 4

Train, Validation, Test, and Data Leakage

In Lesson 2 you saw that training error lies: a model can ace the data it learned from and still flop on new data. So the only honest verdict comes from data the model has never seen. This lesson is about setting that data aside correctly, and the subtle mistakes that secretly let the answer leak in.

Meet Dev, an analyst at a small online lender. He builds a model to predict whether a new loan will default (the borrower stops repaying). His first version scores a thrilling 98% accuracy on held-out loans. Two months after launch, on real applicants, it is no better than guessing. Nothing was broken. One column had been handing the model the answer. By the end of this lesson you will know exactly how that happens, and how to stop it.

You will be able to:

  • Explain why a held-out test set is the only honest stand-in for future, unseen data
  • Split data into train / validation / test and say what each set is for
  • Recognise and prevent data leakage: target, preprocessing, and temporal

Prerequisites: you can run R and read its output, and you know what training error, test error and overfitting are (Lesson 2: The Bias-Variance Tradeoff).

The strip below is Dev's data, cut into the three sets. Flip the switch to sneak in a column built from the answer and watch the test score balloon. We unpack every piece of this over the next few steps.

Why hold data back

The test set is a stand-in for the future

Dev's real goal is not to score well on the 1,000 loans already sitting in his database. Those borrowers' fates are settled. His goal is to predict the next applicant, someone not in the data at all. He cannot test on the future because it has not happened yet, so he does the next best thing: he hides a slice of his current loans, trains as if they do not exist, then unhides them and checks. That hidden slice is the test set, and it role-plays the future.

How honest is the verdict? With \(m\) held-out loans, where \(\hat{y}_i\) is the model's prediction for loan \(i\), \(y_i\) is what actually happened, and \(\mathbf{1}[\cdot]\) equals 1 when the prediction is wrong, the test error rate is

\[ \widehat{\mathrm{Err}} \;=\; \frac{1}{m}\sum_{i=1}^{m} \mathbf{1}\!\left[\hat{y}_i \neq y_i\right]. \]

This is just "the fraction of held-out loans the model got wrong." It is an honest estimate of future error for exactly one reason: the model never saw these rows while learning. The moment it does, the estimate is poisoned.

One slice is not enough, though. Dev also needs to choose between models (which features? how complex?), and every choice he makes by peeking at a set burns that set's honesty. So practice splits the data three ways:

Set Share Its one job When you touch it
Train ~60% Fit the model: learn the coefficients, grow the trees Constantly, while fitting
Validation ~20% Tune and compare: pick features, complexity, settings Repeatedly, while choosing
Test ~20% The final, honest verdict on the chosen model Once, at the very end

Here is that split in R. Each lesson runs in a fresh session, so we build Dev's loans inline, then carve out validation and test before doing anything else.

RInteractive R
set.seed(2026) n <- 1000 loans <- data.frame( income = round(rlnorm(n, log(45000), 0.4)), # annual income, $ loan_amt = round(runif(n, 2000, 40000)), # amount borrowed, $ term_mo = sample(c(12, 24, 36, 48), n, TRUE), # repayment term, months age = round(runif(n, 21, 70)) ) # whether each loan actually defaulted: risk rises with the debt-to-income ratio risk <- plogis(-3.2 + 4.0 * (loans$loan_amt / loans$income)) loans$defaulted <- rbinom(n, 1, risk) set.seed(7) shuffle <- sample(nrow(loans)) test_id <- shuffle[1:200] # 20% held out as the final test val_id <- shuffle[201:400] # 20% for tuning and comparing train_id <- shuffle[401:1000] # 60% to fit on train <- loans[train_id, ] val <- loans[val_id, ] test <- loans[test_id, ] c(train = nrow(train), validation = nrow(val), test = nrow(test)) #> train validation test #> 600 200 200