caret mnLogLoss() in R: Multinomial Log Loss Metric

The mnLogLoss() function in caret is the lightweight summaryFunction that trainControl() calls on every resample when you want to tune a classifier purely on log loss. It accepts a data frame with obs, pred, and one probability column per class, then returns a single named scalar logLoss that train() minimises across the tuning grid. Use it when probability calibration is the only metric that matters and you want to skip the heavier ROC and macro-F1 machinery in twoClassSummary() or multiClassSummary().

mnLogLoss(df, lev = levels(df$obs))                              # direct call
trainControl(summaryFunction = mnLogLoss, classProbs = TRUE)     # wire-in
train(..., metric = "logLoss", maximize = FALSE)                 # tune (smaller is better)
fit$resample[, c("Resample", "logLoss")]                         # per-fold log loss
fit$results[, c("mtry", "logLoss", "logLossSD")]                 # mean and SD per grid row
mnLogLoss(df, lev = levels(df$obs))["logLoss"]                   # scalar extract

What mnLogLoss() does in one sentence

mnLogLoss() is caret's minimal logarithmic-loss scoring contract. caret calls it on each fold's held-out predictions when the outcome is a factor with two or more levels and classProbs = TRUE. The body returns one number, logLoss, equal to the average negative log-likelihood of the true class under each row's predicted probability vector. Rows assigned high probability to the correct class contribute near zero; rows confidently mislabelled contribute large positive values.

mnLogLoss() syntax and arguments

The signature has three arguments and the data frame must carry obs, pred, and one probability column per class. caret fixes the shape so every summaryFunction is interchangeable across binary, multi-class, and regression workflows.

The required argument is data: a data frame with obs (truth factor), pred (predicted factor, required by the contract but unused by the math), and one numeric column per factor level holding class probabilities. Probability column names must match factor levels exactly. train() passes lev automatically; supply it yourself when calling outside the resample loop. The return is a named numeric of length 1; "logLoss" is the only valid metric = value when this summaryFunction is wired in.

mnLogLoss() examples by use case

Three patterns cover the common calls: wire-in for CV, tune on log loss, inspect per-fold variance. Each reuses the same function with metric = "logLoss" and maximize = FALSE.

Each row is the mean across five folds. caret picks the mtry with the lowest log loss because maximize = FALSE flips the optimisation direction. The logLossSD column is the across-fold standard deviation, the right input for a fragility check.

Both models are 100 percent accurate, but the calibrated probabilities score seven times better on log loss. This is the gap AUC and Accuracy hide.

Per-fold rows reveal fragility the averaged row hides. A log loss range from 0.08 to 0.20 across folds is wide for iris, hinting one fold caught the harder versicolor-virginica boundary cases. Report mean and worst-fold together; the worst fold predicts real-world failure modes better than the average.

mnLogLoss() vs alternatives

caret ships five summaryFunctions; mnLogLoss is the smallest. Choose by outcome type and which metric drives the decision.

Pick mnLogLoss when probability calibration is the only metric you tune on. If you also need AUC or per-class F1 alongside, multiClassSummary returns both in one call and saves a second pass. For binary outcomes where threshold choice matters, twoClassSummary returns ROC at the resample level so you can balance Sens and Spec.

Common pitfalls

Four mistakes account for nearly every mnLogLoss() bug. Each has a quick fix.

The fix is to set classProbs = TRUE. Without per-class probability columns, log loss cannot be computed and caret aborts before training starts.

caret picks the worst row instead of the best because maximize = TRUE is the default. Log loss is a loss, not a score, so always pass maximize = FALSE. Metric typos surface as nonsense bestTune rows.

Some models return hard 0 or 1 probabilities, which would otherwise cause log(0) = -Inf. caret silently clips the probability matrix to [eps, 1-eps] before applying log, so the result is a tiny positive number. If your log loss is suspiciously close to zero, check the model is returning real probabilities, not hard labels.

Rename prob1, prob2, prob3 to a, b, c. caret indexes probability columns by level name; mismatched names trigger the missing-columns error. The same trap bites every summaryFunction.

Try it yourself

Related caret functions

The summaryFunction family lives in caret's resampling layer:

• multiClassSummary() for 3+ class outcomes with the full 14-metric scorecard. See caret multiClassSummary() in R.
• twoClassSummary() for binary outcomes with ROC, Sens, Spec. See caret twoClassSummary() in R.
• defaultSummary() for regression resample scoring. See caret defaultSummary() in R.
• postResample() for two-vector scoring outside the resample loop. See caret postResample() in R.
• trainControl() for wiring custom summaryFunctions. See caret trainControl() in R.

For implementation details, see the caret performance documentation.

FAQ

What does mnLogLoss() return?

mnLogLoss() returns a named numeric vector of length 1 with name logLoss. The value is the mean negative log-likelihood of the true class under the predicted probability vector, averaged across rows in the supplied data frame. Lower is better. "logLoss" is the only valid metric = value when this summaryFunction is wired into trainControl(), and you must pair it with maximize = FALSE so train() selects the row that minimises the loss.

How does mnLogLoss handle probabilities of exactly 0 or 1?

caret clips the probability matrix to [eps, 1-eps] before taking logs, where eps is a small machine-precision value. This prevents log(0) = -Inf from poisoning the resample average. A row where the model returned probability 0 for the true class still produces a large but finite contribution. If log loss looks suspiciously close to zero across folds, your model is returning hard 0/1 labels rather than calibrated probabilities.

Does mnLogLoss work for two-class outcomes?

Yes. mnLogLoss accepts any factor with 2 or more levels. For binary problems it computes the same log loss you would get from a manual -mean(y*log(p) + (1-y)*log(1-p)) calculation. The advantage over twoClassSummary is speed: you skip the ROC curve entirely. The trade-off is losing the ROC, Sens, and Spec columns, so reporting requires a second scoring pass.

Should I tune on logLoss or AUC?

Tune on logLoss when calibrated probabilities matter to downstream consumers (pricing models, risk scores, cost-weighted thresholds). Tune on AUC when ranking matters but threshold choice happens later (top-K selection, human review queues). logLoss is calibration-sensitive; AUC is rank-only. Models that ace AUC can have terrible log loss if probabilities are systematically squeezed toward 0.5. For high-stakes projects, fit one model on each metric and compare on a held-out test set.

Why is mnLogLoss faster than multiClassSummary?

multiClassSummary computes 14 metrics per resample; two (AUC, prAUC) load pROC and iterate the probability matrix. mnLogLoss computes one metric in base-R arithmetic with no pROC or ModelMetrics import. On a 5-fold CV the saving is small absolute, but in a 50-fold repeated CV with a Bayesian-optimisation outer loop it compounds.