Navigate
Courses Tutorials Practice Pricing Tools Upgrade

Accuracy metrics: MASE, RMSSE and pinball loss

Today let's learn how to score a forecast honestly, using two months of cone sales from a small boardwalk ice-cream cart.

Pier Nine sells cones from a cart on the boardwalk. Below are all 63 days of its sales, from June 1 to August 2, 2026. Weekdays run close to 15 cones a day, weekends climb to around 34, and every so often a rain washout drops a day down to almost nothing.

The last 14 days are marked apart as the test window. No model gets to see them while it is being built. They only get used afterward, to check how good each forecast really was.

Here is the whole series, with the test window marked apart.

That is the series every metric in this lesson scores against.

Two forecasts to score

Let's build that series for real, and split it the same way: the first 49 days to train two models on, the last 14 held back to test them against.

RInteractive R
# Build Pier Nine's 63 days of cone sales and split them into 49 training days and 14 test days library(tsibble) library(fable) library(fabletools) library(dplyr) set.seed(930) n <- 63 day <- seq(as.Date("2026-06-01"), by = "day", length.out = n) weekend <- as.integer(format(day, "%u")) %in% c(6, 7) rain <- rbinom(n, 1, 0.15) == 1 base_sales <- rpois(n, ifelse(weekend, 34, 15)) rain_sales <- sample(0:2, sum(rain), replace = TRUE) sales <- base_sales sales[rain] <- rain_sales cart <- tsibble(date = day, sales = sales, index = date) train <- cart |> filter(date <= as.Date("2026-07-19")) test <- cart |> filter(date > as.Date("2026-07-19"))

  

  





Two models get fit on those 49 training days. Seasonal_naive is the simplest forecast Pier Nine's owner could make by hand: repeat each day's sales from the same weekday, one week back, since a Monday tends to look like the Monday before it. ETS is fabletools's automatic exponential smoothing model. It looks at the same 49 days and works out its own level and weekly pattern, without being told what either one is.



Fit both, forecast 14 days ahead, and line the first 5 up against what actually happened.




  

    
RInteractive R

    
# Fit a seasonal-naive model and an ETS model on the training days, then forecast 14 days ahead
fit <- train |> model(
  Seasonal_naive = SNAIVE(sales ~ lag("week")),
  ETS = ETS(sales)
)
fc <- fit |> forecast(h = 14)

data.frame(
  date = test$date,
  actual = test$sales,
  Seasonal_naive = round(fc$.mean[fc$.model == "Seasonal_naive"], 1),
  ETS = round(fc$.mean[fc$.model == "ETS"], 2)
)[1:5, ]
#>         date actual Seasonal_naive   ETS
#> 1 2026-07-20      9             10 12.07
#> 2 2026-07-21      9             24 11.19
#> 3 2026-07-22     11              2 15.17
#> 4 2026-07-23     13             20 14.11
#> 5 2026-07-24      0              1  9.40

    

      
      
    

    

  

  





Look at 21 July. Seasonal_naive forecasts 24 cones for a day that sold 9, off by 15. ETS lands much closer, at 11.19. Neither model is perfect on every day. Turning "how far off" into one honest number is the whole point from here.