An observed series as a noisy view of a hidden state
Today let's understand state space models, using one long, real series as the running example throughout.
The series is Nile, one of R's own built-in datasets: the river's annual flow at Aswan, in Egypt, measured every year from 1871 to 1970, a hundred years in all. Each reading is in units of 10^8 cubic metres.
Here is the whole hundred years, plotted in the order the years actually happened.
Look at that line. For its first three decades it swings around a high band, then somewhere near 1898 it drops and spends the rest of the century swinging around a lower band instead.
A real shift in level, not just one bad year
That drop around 1898 could be two different things: a single unusually low year the eye is exaggerating, or a real, lasting change in how much water the Nile carried every year from then on. The only way to tell them apart is to compare the two periods properly, not just look at the line.
Nile still holds all hundred years. Split it at 1898 and average each half.
The 28 years up to 1898 average 1097.75. The 72 years from 1899 average 849.97, a drop of about 248, close to 23 percent of the earlier level. And it is not one bad year dragging that second average down: the drop holds across all 72 of those years, which is what makes it a real shift and not noise.
This matches a changepoint researchers have already found in this exact series, right around 1898 (Cobb, 1978), noted on R's own help page for the Nile dataset.
Writing the observation equation and the state equation
Two equations sit behind everything in this lesson, and Nile already gives every symbol in them a real meaning.
\[ y_t = \mu_t + \epsilon_t \]
This is the observation equation. \(y_t\) is the recorded Nile flow in year \(t\), the actual number in the dataset, the one you plotted and averaged in the last two steps. \(\mu_t\) is the hidden level: the true, underlying flow that year, which nobody ever measures directly. \(\epsilon_t\) is that year's own measurement noise, whatever pushed the recorded flow above or below the true level, purely by chance.
\[ \mu_t = \mu_{t-1} + \eta_t \]
This is the state equation. It says the hidden level this year, \(\mu_t\), equals last year's hidden level, \(\mu_{t-1}\), plus a small nudge, \(\eta_t\). The level is allowed to drift a little every year, but it is never allowed to jump.
Of the four symbols, \(y_t\) is the one you actually observe, sitting right there in the dataset. \(\mu_t\), \(\epsilon_t\) and \(\eta_t\) never show up in the data at all, but \(\mu_t\) is the one this whole lesson is really about: the hidden level that \(\epsilon_t\) and \(\eta_t\) together hide from you.
Put the two equations together and you have the local level model, the simplest state space model there is. Every richer state space model later in this course keeps this same shape, one equation for what you observe and one for how the hidden state moves, and only makes the state itself bigger: a level and a trend, a level and a season, and so on.
Splitting the flow before and after 1898
See the same before-and-after story as a picture, split into its two periods instead of left as one solid line.
Switch between the line view and the point view. Either way, the same two colours mark the same two periods, 28 years above the split and 72 below it, with the whole picture stepping down right at the 1898 boundary you already found in the means.
How far the state equation lets the level move
The state equation, \(\mu_t = \mu_{t-1} + \eta_t\), has one thing you have not seen yet: how big \(\eta_t\), the yearly nudge, actually is. That size decides how closely the fitted level tracks the data it is watching.
Drag this slider and watch that same trade-off play out on a generic curve.
A stiff line barely moves from one point to the next: that is what a state space model with a tiny \(\eta_t\) looks like, since every year's nudge is small. A wiggly line chases every point instead: that is a large \(\eta_t\), since the level is then free to move almost as much as the data itself does.
Nile's own moving averages show the same trade-off in real numbers, with no slider involved.
By 1900 the raw flow has already fallen to 840. The 3-year average is close behind it, at 829.3, because it only looks at three nearby years. The 15-year average is still up at 951.6, because it is still averaging in over a decade of the old, higher years from before 1898.
A state space model with a large \(\eta_t\) behaves like that 3-year average: quick to follow a real shift. One with a small \(\eta_t\) behaves like the 15-year average: slow, still smoothing over a shift that happened years earlier.
Quick check: state change or just noise?
You have now seen the 1898 shift three ways: as two means, as two coloured bands, and as two moving averages that both agree it happened. Which part of the state space model does a change like that actually belong to?
ETS and ARIMA: two routes to the same local level model
The local level model is not just a diagram. Two of R's own forecasting functions estimate it directly, each by a completely different search, and it is worth seeing that they land in nearly the same place.
Fit it the first way, as ETS(A,N,N): additive error, no trend, no season, which is exactly the local level model written in state space form.
report() shows alpha = 0.2455339, the same alpha from the state equation two steps back: how much of last year's surprise gets folded into this year's level. forecast() puts 1971, the year right after the data ends, at 805 on average.
Now fit the same idea a second way.
ARIMA(0,1,1) with no constant differences the series once, which removes \(\mu_t\)'s drift directly instead of estimating it as a separate state, and then fits one moving-average term, theta = -0.7329, to what is left. Its forecast for 1971 comes out at 798.4.
805 against 798 is within about 1 percent, close enough to say the two are clearly describing the same series the same way. And there is a direct link between their two numbers: 1 + theta works out to 0.267, close to alpha's 0.2455. The two are not identical, because ETS and arima() optimise slightly different likelihoods to get there, but they sit close enough to confirm both are circling the same local level model from two different directions.
Estimating a year nobody measured
One practical payoff of writing a series this way: the model still has something to say about a year with no data in it at all. See it by deleting a real stretch of Nile's own history.
1891 to 1910 and 1931 to 1950, 40 years in total, are now missing, following the exact example documented on R's own ?Nile help page. Fit the local level model on this gappy series with StructTS(), then smooth it with tsSmooth().
Even though NileNA has no data at all for 1891 to 1895, tsSmooth() still returns a real number for the level in every one of those years. That is because the state equation, \(\mu_t = \mu_{t-1} + \eta_t\), only ever needs \(\mu_{t-1}\); it does not need \(y_t\) to produce \(\mu_t\). In a year with an observation, the observation equation adds its own information on top of that. In a year with none, \(\mu_t\) simply carries forward from \(\mu_{t-1}\) plus the model's own estimated size of \(\eta_t\), which is why the numbers above decline gently across the whole 20-year gap instead of freezing in place.
Reading the hidden level off as its own number
Fit the same local level model on the whole, ungapped Nile series this time, and look at what the smoothed level does right through the 1898 shift.
This is \(\mu_t\) itself, read straight off the fitted model, one named number for every year: 1078 in 1896, 1038 in 1897, 1000 in 1898, 951 in 1899, and 919 by 1900. It does not jump from the old level to the new one in a single year. It slides down over several years instead, because \(\eta_t\) only ever nudges \(\mu_t\) a little at a time, exactly as the state equation says it must. That is the real difference between reading the raw \(y_t\), which does jump around year to year since it still carries \(\epsilon_t\), and reading \(\mu_t\), which only ever moves as fast as the state equation allows.
Quiz: the state space idea in one pass
The observation and state equations and the ETS/ARIMA fit both point at the same fact from two different angles. Put them together.
Your turn: find the model's estimate for a missing year
sm_na still holds the smoothed level for every year of NileNA, missing years included. Here is what it says for the two years right around 1900, skipping 1900 itself.
1899 sits at 923.2821 and 1901 at 907.1623, both estimated from inside a 20-year gap with no data at all. Now find 1900's own smoothed level the same way you found 1891 to 1895's a little earlier: with window() on sm_na's level column.
Show answer
# Print the smoothed level for 1900 alone
window(sm_na[, "level"], start = 1900, end = 1900)
#> Time Series:
#> Start = 1900
#> End = 1900
#> Frequency = 1
#> [1] 915.2222References
- Durbin, J. and Koopman, S.J. (2001), Time Series Analysis by State Space Methods, Oxford University Press. Source of the Nile series and the local level model this lesson builds from.
- Cobb, G.W. (1978), "The Problem of the Nile: Conditional Solution to a Changepoint Problem", Biometrika 65(2), 243-251. The changepoint near 1898 that the before-and-after comparison earlier in this lesson confirms.
- [R documentation for the
Niledataset](https://stat.ethz.ch/R-manual/R-patched/library/datasets/html/Nile.html) (datasets package). Confirms the 1871-1970 span, the units, and the changepoint citation used in this lesson. - Harvey, A.C. (1989), Forecasting, Structural Time Series Models and the Kalman Filter, Cambridge University Press. The standard reference for the general state space form, of which this lesson's local level model is the simplest case.
- Forecasting: Principles and Practice, chapter 8: Exponential smoothing - Hyndman, R.J. and Athanasopoulos, G. (3rd ed., OTexts). The state space formulation behind the
ETS()fit in this lesson.
What you can do now
You can now say what a state space model actually claims: an observation equation for the number you see, plus noise, sitting on a state equation for how a hidden level moves underneath it. On Nile, that hidden level held near 1098 for 28 years, then slid down to about 850 for the rest of the century, and \(\mu_t\), not \(\epsilon_t\), is what carried it there.
You have also seen the same local level model twice, from two different directions: ETS(A,N,N) and ARIMA(0,1,1) landing within about a percent of each other on the same series, because they are fitting the same pair of equations two different ways.
And you have used two things that fall straight out of writing a series this way: a real number for a year with no observation at all, and a hidden level you can read off directly, year by year, instead of only ever seeing it through the noise sitting on top of it.
Next, you will open up exactly how a state space model computes all of this: the predict-then-update loop that produced every smoothed number in this lesson, worked out by hand on a few years of data.