Grouped, Blocked, and Time-Aware CV

In Lesson 1 you took the luck out of a model's score: k-fold cross-validation rotates the holdout so every row is tested once, then averages for a steady number you can trust.

That trick has a hidden assumption, and when it breaks it breaks quietly. Meet Bean Theory, a small coffee chain. Their model predicts a café's daily cups sold, and 5-fold cross-validation gave it a rock-steady error of about 4.7 cups. Everyone was happy, until they opened a new café and the same model was off by 15. Nothing was wrong with the model. Something was wrong with the folds.

By the end of this lesson you will be able to:

• Explain why random k-fold reports a dishonest, too-good score when rows are grouped or time-ordered
• Build grouped folds that keep each group (each café) entirely in train or validation
• Build time-aware folds that always train on the past and validate on the future

Prerequisites: you finished Lesson 1 on cross-validation, you can fit a model with lm() and call predict(), and you have met the idea of data leakage. Flip the switch below to see leakage in action, the exact trap this lesson defuses.

The hidden assumption in k-fold

k-fold cross-validation shuffles the rows and deals them into folds at random. That shuffle only makes sense if the rows are independent and interchangeable: any row is as good as any other to hold out, and knowing one row tells you nothing special about another.

Huge amounts of real data quietly violate that. Two patterns come up again and again:

• Grouped rows. Many rows belong to the same underlying thing: 20 days of sales from the same café, 12 visits from the same patient, 100 clicks from the same user. Rows inside a group are alike, so a random shuffle scatters copies of the same group across every fold.
• Time-ordered rows. Rows arrive in sequence, and the future depends on the past: yesterday's sales predict today's. A random shuffle mixes future and past together, letting the model learn from days that, in real life, had not happened yet.

The widget below is ordinary random k-fold: it deals the rows into folds and rotates which one is held out. It is exactly the right tool when rows really are interchangeable, and exactly the wrong one when they are not. Watch it rotate, then hold that picture in mind as we feed it data that is not interchangeable.