Navigate
Courses Tutorials Practice Pricing Tools Upgrade

Judgmental adjustments and forecast governance

Today let's understand when a person should reach in and change a statistical forecast by hand, and when that same habit is quietly making the forecast worse.

Priya is the demand planner for a company that sells insulated steel water bottles online. Every month a statistical model forecasts next month's unit sales for one product line, and before that number goes to the factory, Priya can adjust it herself. Over the last 12 months she adjusted it 6 times.

Here is her whole year: the statistical forecast, the adjusted forecast she actually sent to the factory, and what customers actually bought.

Look at where the adjusted forecast, the line Priya actually sent to the factory, pulls close to the actual line in some months and pulls away from it in others. That gap, and why it opens up only sometimes, is what this lesson explains.

What counts as a judgmental adjustment

A judgmental adjustment is a change a person makes to a model's forecast, by hand, before that forecast gets used. The statistical forecast is whatever the model computed from past sales. The adjustment is the amount Priya added or subtracted. The adjusted forecast is the statistical forecast plus the adjustment, and that adjusted number, not the model's own number, is what actually goes to the factory.

Build Priya's 12-month ledger: the statistical forecast, the adjustment, the adjusted forecast, and what customers actually bought.

RInteractive R
# Build Priya's 12-month ledger: forecast, override, adjusted forecast, and actual ledger <- data.frame( month = 1:12, statistical_forecast = c(900, 950, 1000, 1100, 1150, 1250, 1300, 1350, 1300, 1400, 1450, 1500), adjustment = c(0, 0, 150, 220, 0, 130, -100, 0, 200, 90, 0, 0), reason_category = c("none", "none", "known_one_off", "optimism", "none", "known_one_off", "anchoring", "none", "authority", "known_one_off", "none", "none"), reason_detail = c("", "", "Confirmed bulk order from a corporate gifting client", "Regional manager pushed for a stretch target, no account named", "", "Scheduled promotion confirmed for the month", "Planner leaned on last July's figure instead of updating for a higher baseline", "", "VP demanded a bigger number before a board meeting, no account named", "Confirmed reorder from an existing retail partner", "", ""), actual = c(920, 920, 1140, 1080, 1190, 1395, 1330, 1300, 1280, 1510, 1475, 1465), stringsAsFactors = FALSE ) ledger$adjusted_forecast <- ledger$statistical_forecast + ledger$adjustment ledger[, c("month", "statistical_forecast", "adjustment", "adjusted_forecast", "actual")] #> month statistical_forecast adjustment adjusted_forecast actual #> 1 1 900 0 900 920 #> 2 2 950 0 950 920 #> 3 3 1000 150 1150 1140 #> 4 4 1100 220 1320 1080 #> 5 5 1150 0 1150 1190 #> 6 6 1250 130 1380 1395 #> 7 7 1300 -100 1200 1330 #> 8 8 1350 0 1350 1300 #> 9 9 1300 200 1500 1280 #> 10 10 1400 90 1490 1510 #> 11 11 1450 0 1450 1475 #> 12 12 1500 0 1500 1465

  

  





In 6 of the 12 months, the adjustment is 0, meaning Priya left the model's number untouched. In the other 6, she changed it, by as little as 90 units and as much as 220.



Now measure how those 6 changes did overall, using mean absolute error, MAE: the average size of a forecast's miss against the actual, ignoring whether it missed high or low.




  

    
RInteractive R

    
# Mean absolute error for the unadjusted and the adjusted forecast, over all 12 months
mae <- function(forecast, actual) mean(abs(forecast - actual))

unadjusted_mae <- mae(ledger$statistical_forecast, ledger$actual)
adjusted_mae   <- mae(ledger$adjusted_forecast, ledger$actual)

round(c(unadjusted_mae = unadjusted_mae, adjusted_mae = adjusted_mae), 2)
#> unadjusted_mae   adjusted_mae
#>          55.42          69.58

round(100 * (adjusted_mae - unadjusted_mae) / unadjusted_mae, 1)
#> [1] 25.6

    

      
      
    

    

  

  





The statistical forecast alone misses by 55.42 units a month on average. The adjusted forecast, the one Priya actually sent, misses by 69.58, which is 25.6% worse. Read only this one number, and Priya's adjustments look like they made things worse across the board.



Lay the whole ledger out as a table, with each override's reason attached.




Every override in this table has a reason attached. But not every reason points to the same kind of thing: some name a fact the model could not see, and some do not.