How a Temporal Fusion Transformer forecasts
Today let's see what a Temporal Fusion Transformer actually looks at before it makes a forecast, and why it can end up paying more attention to a month from a year ago than to last month.
Suppose you handle demand planning for liquor retailing in four Australian states: New South Wales, Victoria, South Australia and Western Australia. Across the 37 years in this data, December's turnover jumps well above November's every single year: by 42 percent in New South Wales, and by almost 60 percent in South Australia. So the calendar alone already carries real information about what is coming.
We will forecast December 2018 for all four states using only the data through November 2018. The real December 2018 figures stay hidden until the end, so we can check the forecast against them once it is made.
The chart below plots each state's monthly turnover from December 2017 to December 2018, in millions of Australian dollars.
Look at where each line peaks: at both ends of the chart, December 2017 and December 2018, every single state sits at its highest point of the year.
Three kinds of input: static, known-future, observed-past
A Temporal Fusion Transformer does not treat every piece of information about a forecast the same way. It sorts each input into one of three kinds, and keeping them separate is the whole point of the architecture.
The first kind is the static covariate: a fact about the series that never changes over time. Here that is the state itself. New South Wales simply sells far more liquor than South Australia every month of the year, and that gap holds in December just as much as it holds in June.
The second kind is the known-future input: something you already know before the forecast is even made, because it comes from the calendar rather than from a measurement. For this backtest, the target month is December 2018, and whether that month is December is settled in November, long before anyone measures what actually sold.
The third kind is the observed-past input: a value that was only available up to the moment you are forecasting from. November 2018's turnover is observed-past. It is a real number, not a guess, but it stops the moment the forecast begins.
The table below lines up all four states against their static, known-future and observed-past inputs for this one backtest.
Treat the known-future input the same way you would an observed-past one, or drop it from the model altogether, and you throw away the one piece of information that is free here: you already know, for certain, that the month being forecast is December.