Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 5 of 7

Scoring Rules and Regression Metrics

In Lesson 4 you tuned a model until its error was as low as it would go. But low error on which ruler? This lesson is about the rulers themselves, and a fact that catches most people off guard: change the metric and the "best" model can change with it.

Meet Priya, who runs a bakery and has to forecast next week's daily cake order. Two forecasters are up for the job: Priya's own steady rule, and her data-science-student nephew Sam's bolder model. On the very same week, one metric will crown Sam and another will crown Priya. By the end you will know exactly why, and which ruler to trust for which decision.

By the end you will be able to:

  • Compute RMSE, MAE and MAPE in R and say what behaviour each one rewards
  • Score a probability forecast with log loss and Brier, and see why a confident wrong call is ruinous
  • Define a proper scoring rule and pick the metric that matches your decision

Prerequisites: you can run R and index a data frame, and you have tuned a model to a low error in Lesson 4. A cross-validation fold from Lesson 1 helps, but is re-explained where it is needed.

The job of a metric

One number for a whole week of misses

A metric takes a whole column of forecasting misses and squeezes it into a single number you can compare. Before we can argue about which number is best, we need the raw material: what each forecaster predicted, and what actually happened.

Each lesson runs in a fresh R session, so let us build Priya's week right here, along with both forecasts. Sam's model matched the six ordinary days almost perfectly, then whiffed badly on Sunday. Priya was a few cakes off every single day.

RInteractive R
week <- data.frame( day = c("Mon","Tue","Wed","Thu","Fri","Sat","Sun"), actual = c(60, 55, 62, 58, 64, 120, 66), # cakes actually sold priya = c(65, 50, 67, 53, 69, 115, 72), # Priya's steady forecast sam = c(60, 55, 60, 58, 64, 120, 36) # Sam's bold forecast ) week$err_priya <- week$actual - week$priya # a forecast miss, in cakes week$err_sam <- week$actual - week$sam week #> day actual priya sam err_priya err_sam #> 1 Mon 60 65 60 -5 0 #> 2 Tue 55 50 55 5 0 #> 3 Wed 62 67 60 -5 2 #> 4 Thu 58 53 58 5 0 #> 5 Fri 64 69 64 -5 0 #> 6 Sat 120 115 120 5 0 #> 7 Sun 66 72 36 -6 30

  

  





The error on day \(i\) is just actual minus forecast, \(e_i = y_i - \hat{y}_i\), where \(y_i\) is what Priya really sold and \(\hat{y}_i\) is what the forecaster predicted. Sam's errors are almost all zero, except a whopping 30 on Sunday. Priya's are a steady 5 or 6 every day. Every metric ahead is just a different way of boiling those seven errors down to one score.