Nested Cross-Validation

You run a small bakery and want to predict tomorrow's cake sales from the day's high temperature. You try eight candidate models, use cross-validation to score each, and keep the best: it lands 7.7 cakes off, on average. You quote that number to your supplier. The trouble is you picked that model because it scored best on those exact folds, so 7.7 is the luckiest of eight tries, not the honest truth. This lesson shows you how to get a number you can actually stand behind.

By the end you will be able to:

• Explain why tuning and scoring on the same cross-validation flatters your accuracy
• Set up nested cross-validation: an outer loop that grades, wrapping an inner loop that tunes
• Say exactly what the outer scores estimate, and how you still choose the final model to ship

Prerequisites: Lesson 1 (k-fold cross-validation), Lesson 2 (grouped and time-aware CV), and you know that "tuning" means trying several settings of a knob and keeping the best. You can fit lm() and call predict().

One cross-validation cannot both choose and grade

In Lesson 2 you plugged the leak where a careless split lets test rows sneak into training. There is a subtler leak left, and it does not come from the split at all. It comes from choosing.

Here is the whole problem in one sentence: if you use one cross-validation to pick the best model AND to report its score, that score is too good. Picking the winner is itself a decision made from the folds, so the winner's fold score is partly real skill and partly the luck that made it come out on top this time.

An everyday version: let 20 students each guess tomorrow's temperature, then crown whoever guessed closest and announce "my best student predicts within 1 degree." Tomorrow that same student is off by 6. You never measured the student's skill. You measured the luck of picking the best of 20 after seeing the answer.