Fitting a dynamic harmonic regression

Today let's understand how to model a season that repeats once a year, using a handful of sine and cosine terms inside an ARIMA fit, on a real weekly electricity demand series.

Here's the running example. Victoria, an Australian state, has to keep its electricity grid supplied every week of the year, and demand rises and falls with the seasons: more air conditioning in summer, more heating in winter. We'll use 156 consecutive weeks of the grid's actual electricity demand, in gigawatt-hours (GWh), covering 2012 through 2014, so the yearly rhythm repeats three times over.

Plot the week number against that week's demand, one point per week.

The line climbs and falls roughly once a year, from a low near 632 GWh to a high just over 1,004 GWh. But look closely: it isn't one smooth, even wave. Some years bend earlier or later than others, and there are bumps along the way instead of one clean sine curve.

Why can't seasonal ARIMA handle a 52-week season?

Seasonal ARIMA adds a seasonal part on top of the ordinary ARIMA(p, d, q): a piece written (P, D, Q)[m], with P seasonal AR terms, D seasonal differences, Q seasonal MA terms, all built around a season of length m, the number of periods before the pattern repeats.

A seasonal difference at lag m compares each observation to the one exactly m periods earlier. For monthly data with a yearly season, m = 12, and the comparison is this January against last January. Here the season is a year of weekly data, so m = 52: this week against the same week last year.

Here's the problem. That single seasonal difference has nothing to compare the first m weeks against, since there's no earlier year for them to look back to, so it throws those weeks away before a single residual exists.

See how many of the 156 weeks would actually be left after one seasonal difference, at a few different values of m.

See that same loss expressed as a percentage of the 156 weeks.

If this same 156-week series had a season of only 12 weeks or 4 weeks, one seasonal difference would cost a small, forgettable slice of it, 7.7% or 2.6%. But at m = 52, that same single difference costs 33.3% of the series, a full third, before fitting anything. It isn't seasonal differencing itself that's the problem. It's a season this long relative to how much data there is.