Cross-Validation Strategies
You already know the golden rule of honest evaluation: never judge a model on the data it trained on. Hold out a test set, and its score stands in for how the model will do on the future. But there is a crack in that plan.
Sam, an analyst at a used-car marketplace, is checking a model that predicts a car's fuel economy from its weight and horsepower. He holds out a quarter of his cars, scores the model, and gets a typical error of 3.3 mpg. He reshuffles, holds out a different quarter, and gets 1.6 mpg. Same cars, same model, and the verdict just doubled. Which number does he report? Cross-validation is the fix, and this lesson builds it straight out of that problem.
By the end you will be able to:
- Explain why a single train/test split gives a luck-of-the-draw score
- Run k-fold cross-validation and read its result
- Weigh LOOCV and repeated k-fold, and choose a sensible number of folds
Prerequisites: you can fit a model with lm() and call predict(), and you know what a held-out test set is (Lesson: Train, Validation, Test, and Data Leakage).
One split is a coin flip
Let us watch Sam's problem happen for real. The data is mtcars: 32 cars that Motor Trend road-tested in 1974, each with its fuel economy (mpg), weight (wt, in thousands of pounds) and horsepower (hp).
Sam holds out 8 of the 32 cars (a quarter), fits an ordinary regression on the other 24, and measures the typical prediction error on the 8 he held out. That error is the RMSE (root-mean-square error), reported in mpg: an RMSE of 2.5 means the model is off by about 2.5 mpg on a typical car. Then he does it again with a different random 8, and again, six times over:
Every number came from the same 32 cars and the same model. The only thing that changed was which 8 cars happened to land in the hold-out set, yet the estimated error swings from 1.60 to 3.28 mpg, more than double. With only 8 cars deciding the score, one unlucky pair of gas-guzzlers can wreck it. A single split does not really measure the model; it measures the model plus the luck of the draw.
Why did the score change?
Sam ran the exact same model on the exact same 32 cars six times and got six different error estimates, from 1.60 to 3.28 mpg. What is the main reason the number moved?
Rotate the holdout: k-fold cross-validation
The wobble comes from letting one small hold-out set decide everything. So do not let it. k-fold cross-validation splits the cars into k equal groups, called folds. Then it takes turns: fold 1 is held out while folds 2 to k train the model and score it; then fold 2 is held out while the rest train; and so on, until every fold has been the hold-out exactly once. That gives k scores, and their average is the cross-validated error.
The beauty is that nothing is wasted and nothing is special. Every car trains the model in k - 1 of the rounds and tests it in exactly one. No single lucky or unlucky split can dominate, because you average over all of them.
Step through the folds below. Watch the hold-out fold (highlighted) rotate through the data while the rest train, and see the k fold-scores collapse into one steadier average.
The cross-validated error, precisely
Here is that idea written exactly. Number the rows \(1\) to \(n\) (for Sam, \(n = 32\) cars). Split them into \(k\) folds \(F_1, F_2, \dots, F_k\) of roughly equal size, so \(|F_j|\) is the number of cars in fold \(j\).
For each fold \(j\), train the model on every car except those in \(F_j\); call that fitted model \(\hat f_{-j}\), where the \(-j\) means "fold \(j\) left out." Score it on the held-out fold with the root-mean-square error
\[ \mathrm{RMSE}_j \;=\; \sqrt{\frac{1}{|F_j|}\sum_{i \in F_j}\big(y_i - \hat f_{-j}(x_i)\big)^2}, \]
where \(y_i\) is car \(i\)'s true mpg, \(x_i\) its weight and horsepower, and \(\hat f_{-j}(x_i)\) the prediction from the model that never saw car \(i\). The cross-validated error is simply the average of the \(k\) fold scores:
\[ \mathrm{CV}_k \;=\; \frac{1}{k}\sum_{j=1}^{k}\mathrm{RMSE}_j. \]
Build 5-fold CV by hand
Let us turn that formula into code and pin down Sam's real number. The setup gives each of the 32 cars a random fold label from 1 to 5. Inside the loop, the held-out fold is fold == f; the training set is everything else. Fill in the blank so the model trains on the other four folds.
Show answer
n <- nrow(mtcars)
set.seed(7)
k <- 5
fold <- sample(rep(1:k, length.out = n))
scores <- numeric(k)
for (f in 1:k) {
train <- mtcars[fold != f, ]
valid <- mtcars[fold == f, ]
fit <- lm(mpg ~ wt + hp, data = train)
scores[f] <- rmse_of(valid$mpg, predict(fit, valid))
}
round(scores, 2) # one RMSE per fold
#> [1] 3.73 1.79 2.61 2.67 3.07
round(mean(scores), 2) # the cross-validated RMSE
#> [1] 2.77Read the two lines together. The five folds individually score anywhere from 1.79 to 3.73 mpg, the same kind of wobble a single split showed. But their average, 2.77 mpg, is one steady estimate that used every car for both training and testing. Sam finally has a number he can report.
LOOCV: hold out just one car
What if you push k as far as it will go, all the way to k = n? Then each fold is a single car: train on the other 31, predict that one, record the error, and repeat 32 times. This is leave-one-out cross-validation (LOOCV), the k = n extreme. Because each model trains on almost all the data (31 of 32 cars), LOOCV wrings the most out of a small dataset.
That is 2.78 mpg, right next to the 2.77 the 5-fold run gave. But notice the cost: LOOCV refit the model 32 times, once per car. With 32 cars that is nothing; with 320,000 rows it is 320,000 fits.
For a plain linear model there is a lovely shortcut that needs no refitting at all. Fit once, then car \(i\)'s leave-one-out residual is its ordinary residual \(e_i\) divided by \(1 - h_{ii}\), where \(h_{ii}\) is the car's leverage (how unusual its weight and horsepower are, read straight off the fit with hatvalues()):
\[ e_{i}^{\,\text{LOO}} \;=\; \frac{e_i}{1 - h_{ii}}. \]
The same 2.78, from a single fit. The shortcut is exact for linear models; most other models have no such trick and really must refit n times.
Is LOOCV always the best choice?
A colleague argues: "LOOCV trains on almost all the data every time, so it is nearly unbiased. It must give the best error estimate, always use it." Where does that reasoning fall short?
Repeated k-fold cross-validation
k-fold already tamed the wild single-split swing. But one small thing still nudges the answer: the random assignment of cars to folds. Shuffle the cars into folds a different way and you get a slightly different \(\mathrm{CV}_k\). Watch:
Three different fold shufflings, three answers from 2.37 to 2.84. Much smaller than the single-split swing, but still there. Repeated k-fold removes it the same way k-fold removed the single-split luck: run the whole k-fold procedure many times, each with a fresh shuffling, and average.
The average over 50 runs, 2.68 mpg, barely changes if you do it all again: the spread between individual runs is only 0.125 mpg. Repeated k-fold earns its keep on small datasets like this one, where a single shuffling can still swing the estimate.
Choosing k: the estimate's own tradeoff
You now have three schemes, 5-fold, 10-fold, and LOOCV. Which k should you actually pick? The surprising answer is that the error estimate itself has a bias and a variance, and \(k\) trades one against the other.
Bias of the estimate. With \(k\) folds, each model trains on a fraction \((k-1)/k\) of the data. Small \(k\) means smaller training sets: 2-fold trains on only half the cars. If the model would clearly do better with more data, its cross-validated error comes out slightly pessimistic, an over-estimate, because every fold's model was handicapped. As \(k\) rises toward \(n\) (LOOCV), each model trains on nearly the full dataset, so that bias shrinks almost to nothing.
Variance of the estimate. Here it flips. \(\mathrm{CV}_k\) is an average of \(k\) fold scores. For an average of \(k\) scores that each have variance \(\sigma^2\) and share an average pairwise correlation \(\rho\),
\[ \mathrm{Var}\!\left(\mathrm{CV}_k\right) \;=\; \frac{\sigma^2}{k} \;+\; \frac{k-1}{k}\,\rho\,\sigma^2. \]
Read the two terms. The first, \(\sigma^2/k\), is the familiar "averaging shrinks noise" term. The second is the catch: when the fold scores are correlated (\(\rho > 0\)), averaging cannot drive the variance to zero. LOOCV is the worst case, its \(n\) models are each trained on datasets that differ by a single row, so they are almost identical and their scores are highly correlated (\(\rho\) close to 1). The second term takes over and the estimate stays noisy, no matter how large \(n\) is. A moderate \(k\) keeps the folds more different from one another, so \(\rho\) is lower and the average is steadier.
Put the two ideas together with the cost of running it:
| Scheme | Data each model sees | Bias of estimate | Variance of estimate | Cost |
|---|---|---|---|---|
| Small k (2 to 3) | little (1/2 to 2/3) | higher (pessimistic) | lower | cheap (k fits) |
| k = 5 or 10 | most (80 to 90%) | low | low | moderate |
| LOOCV (k = n) | nearly all | lowest | higher | expensive (n fits) |
One honest caveat, plain in Sam's own numbers: for a stable model with enough data, the schemes barely disagree (you will see all three side by side in a moment). The tradeoff above bites hardest when data is scarce or the model is flexible and unstable, exactly when getting the estimate right matters most.
Which scheme fits the job?
One teammate is tuning a slow-to-train gradient boosting model on 200,000 rows and needs a trustworthy error estimate without waiting all week. Another has a tiny pilot dataset of 45 patients and one quick logistic model. Which resampling choice suits each?
Putting it to work, and where it breaks
A small reusable helper turns cross-validation into a one-liner. Here is the whole family on Sam's cars, side by side:
Three honest estimates, all within about 0.2 mpg of each other, just as the last step promised for a stable model with enough data. In everyday work you rarely hand-roll the loop: rsample::vfold_cv() builds the folds for you, and caret or tidymodels run the whole resample-and-score cycle, but the loop above is exactly what they do underneath.
A few rules of thumb to carry away:
- Default to 5- or 10-fold. It is the bias-variance-cost sweet spot for almost every problem.
- Repeat it on small data. A few repeats of 10-fold steady the estimate when a single shuffle still swings it.
- Reach for LOOCV only when data is precious (you cannot spare a 20% fold) or the model has a cheap leave-one-out shortcut, like the linear one you saw.
- Stratify for classification, so every fold keeps the same class balance, which is vital when the positive class is rare.
References
A few authoritative places to take this further:
- An Introduction to Statistical Learning, ch. 5 (free PDF) - the clearest first treatment of the validation set, LOOCV, and k-fold, including why LOOCV is higher-variance than k-fold.
- The Elements of Statistical Learning, ch. 7.10 (free PDF) - cross-validation inside the full model-assessment picture, with the bias-variance-of-the-estimate argument in detail.
- Kohavi (1995), A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection - the classic empirical study that made stratified 10-fold the default.
- rsample (tidymodels) documentation - the R tooling for folds:
vfold_cv,loo_cv, and repeated CV, so you never hand-roll the loop in production.
Lesson 1 complete
You can now explain why a single train/test split is a coin flip, run k-fold cross-validation to turn k honest hold-out scores into one steady estimate, weigh LOOCV and repeated k-fold, and choose \(k\) from the bias-variance-cost tradeoff of the estimate itself. Sam has a number he can trust: about 2.7 mpg, not a range from 1.6 to 3.3.
Next, Lesson 2: Grouped, Blocked, and Time-Aware Cross-Validation. Every scheme here assumed the rows were independent and shuffled them freely. When cars share a factory, patients repeat visits, or readings march through time, that shuffle leaks the answer across the split. You will learn to resample so the estimate stays honest even when the rows are not.