caret defaultSummary() in R: Regression Resample Metrics
The defaultSummary() function in caret is the regression-side summaryFunction that trainControl() calls on every resample. It accepts a data frame with columns named obs and pred and returns a named numeric vector of RMSE, R-squared, and MAE. caret wires it in by default for numeric outcomes, so every regression resample row in fit$resample comes from this one function.
defaultSummary(data.frame(obs = y, pred = yhat)) # regression metrics trainControl(summaryFunction = defaultSummary) # explicit default train(..., metric = "RMSE") # optimise on RMSE train(..., metric = "Rsquared", maximize = TRUE) # optimise on R-squared defaultSummary(data, lev = NULL, model = NULL) # full API signature fit$resample # rows defaultSummary produced defaultSummary(df)["MAE"] # extract one metric
Need explanation? Read on for examples and pitfalls.
What defaultSummary() does in one sentence
defaultSummary() is caret's regression scoring contract. It is the function train() calls on each fold's held-out predictions when the outcome is numeric, and the function any custom summaryFunction must imitate in shape. The body simply wraps postResample(data$pred, data$obs) and returns a length-three named numeric vector with RMSE, Rsquared, and MAE.
The data-frame interface (rather than two vectors) exists so caret can pipe in resample frames that may carry extra columns (rowIndex, Resample, model-specific predictions) without breaking the contract.
postResample()'s numeric branch but the columns are wrong. For classification, switch to twoClassSummary or multiClassSummary inside trainControl().defaultSummary() syntax and arguments
The signature has three arguments but only one carries data. caret fixes the shape so any function plugged into summaryFunction is interchangeable.
The required argument is data, a data frame with at minimum columns obs (truth) and pred (model output). The other two, lev (factor levels) and model (training method name), are part of the API contract for swap-ability but are ignored by this implementation. Custom summaryFunctions for classification, like twoClassSummary, read both.
The return value is a flat named numeric vector. train() rbinds one row per resample into fit$resample, then averages columns into fit$results. The names you return become the column names; whichever name matches metric = in the train() call is used for tuning.
rowIndex and Resample. defaultSummary ignores everything except obs and pred, so wrapping it in a custom function and adding metrics is safe.defaultSummary() examples by use case
Four patterns cover almost every call: direct scoring, wiring into trainControl, optimising on a chosen metric, and wrapping it for custom output. Each reuses the same function with slightly different framing.
The same numbers come out of postResample(pred, obs). The reason to use defaultSummary here is consistency: a script that scores resamples via summaryFunction and ad-hoc test sets the same way is easier to maintain.
Wiring defaultSummary into trainControl() is the canonical use. caret defaults to it for numeric outcomes, but stating it explicitly documents intent.
The metric = argument of train() picks which column drives tuning. Pass "RMSE" (minimised) or "Rsquared" with maximize = TRUE.
A fourth pattern wraps defaultSummary to add extra metrics without rewriting the regression core. Bind the new metric to the named vector and keep the original columns intact.
The wrapper keeps the standard three metrics and adds Mean Absolute Percentage Error. Any function that returns a named numeric vector is a valid summaryFunction; using defaultSummary as the base makes the new metric drop into reporting code that already reads RMSE, Rsquared, and MAE.
fit$resample once during model development. It shows per-fold variance that the averaged fit$results row hides. A model with mean RMSE 2.5 and per-fold RMSE ranging 1.8 to 3.6 is less reliable than one with mean 2.7 and range 2.5 to 2.9.defaultSummary() vs alternatives
Caret ships five summaryFunctions plus the postResample helper. The right one depends on outcome type and which metrics you want recorded.
| summaryFunction | Outcome type | Returned metrics | Picks when |
|---|---|---|---|
defaultSummary |
Numeric (regression) | RMSE, Rsquared, MAE | Standard regression resampling (the default) |
twoClassSummary |
Two-class factor | ROC, Sens, Spec | Binary classification with classProbs=TRUE |
multiClassSummary |
Multi-class factor | Accuracy, Kappa, per-class metrics | More than two classes |
mnLogLoss |
Two- or multi-class | logLoss | Probability-calibrated classifiers |
prSummary |
Two-class factor | AUC, Precision, Recall, F | Imbalanced classification, precision-recall |
postResample |
Either | RMSE/Rsq/MAE or Accuracy/Kappa | Quick scoring of two vectors outside train() |
Pick defaultSummary when the outcome is numeric and you want the three standard regression metrics. Switch to twoClassSummary the moment the outcome becomes a binary factor; remember to also set trainControl(classProbs = TRUE, summaryFunction = twoClassSummary). Use multiClassSummary for three or more classes, mnLogLoss when probability calibration matters more than the hard label, and prSummary when classes are imbalanced and ROC overstates performance.
Common pitfalls
Three mistakes account for most defaultSummary() bugs. Each has a quick fix.
defaultSummary reads data$obs and data$pred literally. Rename columns before calling, or use postResample(pred, obs) when starting from vectors. The error message refers to NULL - NULL because the missing columns become NULL.
defaultSummary is regression-only. A factor outcome falls through the same code path but the numeric arithmetic produces NAs. Swap to twoClassSummary or multiClassSummary for factor outcomes, and remember to set classProbs = TRUE when the metric needs probabilities.
Here caret silently swaps in a classification summary because the outcome is a factor; the explicit defaultSummary would have failed earlier. The fix is to make summaryFunction explicit (summaryFunction = multiClassSummary) so the metric set is intentional and the code reads its own intent.
Try it yourself
Try it: Build a 5-fold CV pipeline on airquality (drop NAs) predicting Ozone from Solar.R and Wind with lm. Wire defaultSummary explicitly into trainControl(). Save the mean RMSE to ex_rmse.
Click to reveal solution
Explanation: na.omit() removes rows with missing values; trainControl(summaryFunction = defaultSummary) makes the regression scorer explicit; averaging fit$resample$RMSE gives the cross-validated RMSE that fit$results would also report.
Related caret functions
The metric machinery sits one call away:
postResample()for vector-in scoring outside the resample loop. See caret postResample() in R.trainControl()for swapping summaryFunctions and configuring resamples. See caret trainControl() in R.train()for the resample-and-tune driver that calls defaultSummary. See caret train() in R.confusionMatrix()for the full classification scorecard. See caret confusionMatrix() in R.createDataPartition()for the train/test split that feeds resampling. See caret createDataPartition() in R.
For the upstream reference, see the caret package documentation.
FAQ
What does defaultSummary() return?
For a data frame with numeric obs and pred columns, defaultSummary() returns a length-three named numeric vector: RMSE, Rsquared (squared Pearson correlation), and MAE. caret rbinds one such row per fold into fit$resample, then averages columns into fit$results. The names of the vector become the column names you can pass to metric = in train().
How is defaultSummary() different from postResample()?
postResample(pred, obs) takes two vectors; defaultSummary(data) takes a data frame with columns named obs and pred. defaultSummary calls postResample internally on those columns. The data-frame interface exists so caret can hand the resample frame to any pluggable summaryFunction without unpacking it, and it ignores extra columns like rowIndex and Resample.
Can I use defaultSummary() for classification?
No. defaultSummary is regression-only and returns NAs for factor obs and pred. Use twoClassSummary for binary outcomes (returns ROC, Sens, Spec; requires classProbs = TRUE), multiClassSummary for three or more classes, or mnLogLoss when probability calibration matters. All three share the (data, lev, model) signature so they swap into trainControl(summaryFunction = ...) cleanly.
How do I add a custom metric to defaultSummary's output?
Write a wrapper function with the same (data, lev = NULL, model = NULL) signature that calls defaultSummary(data), computes the extra metric, and concatenates it onto the returned vector with c(base, NewMetric = value). Pass that wrapper into trainControl(summaryFunction = ...). The new column appears in fit$resample and can be passed to metric = in train().
Why is my Rsquared near 1 but predictions look wrong?
caret's Rsquared is the squared Pearson correlation between obs and pred, not the traditional 1 minus SS_res over SS_tot. A model whose predictions are perfectly correlated but shifted by a constant returns Rsquared = 1 while RMSE is large. Always pair Rsquared with RMSE in reporting, and use yardstick::rsq_trad() if you need the traditional definition.