Estimating error by leaving one out
Today let's look at a way to measure a statistic's error without a formula for it at all: drop one observation, recompute the statistic, and do that once for every observation you have.
Nine calls hit an API during a load test, and their round-trip times, in milliseconds, were logged as they came back: 42, 39, 45, 41, 44, 40, 43, 42, 95. Eight of them sit between 39 and 45. The ninth, 95, came back after the server paused for garbage collection.
The strip below stands for those nine calls, one badge per call, numbered in the order just listed. Press "Draw again" and watch R resample all nine with replacement: some calls land more than once in a draw, and some get left out entirely.
That kind of random resampling is one way to measure how much a statistic can vary. Today's method is different. It removes exactly one call at a time, in a fixed order, with no randomness at all.
The jackknife: recompute the statistic, one observation short
The method you just watched pick calls at random has a name: the bootstrap. The jackknife works differently. Instead of resampling at random, it removes exactly one observation, recomputes the statistic on what is left, puts that observation back, and moves to the next one.
Do that once for every observation and you get as many recomputed values as you started with. For the nine response times, that means computing the mean nine times, each time with one call left out.
theta_hat, the mean of all nine response times, is 47.8889. theta_i is built by sapply(1:n, function(i) mean(x[-i])): for each position i from 1 to 9, it drops call i and averages the eight that are left.
Walk through the first entry by hand. Drop call 1, the 42 ms response, and average the remaining eight response times: 39, 45, 41, 44, 40, 43, 42, 95. That comes out to 48.625, the first number in theta_i. Doing that eight more times, once per call, produces the full set of nine values above.
Call those nine values the jackknife replicates of the mean: one number for every observation, each one the mean of the data with exactly that observation missing.
Notice the last entry, 42.000. That is what happens when you drop call 9, the 95 ms response: the remaining eight ordinary response times average to 42, well below theta_hat's 47.8889. Every other drop barely moves the mean at all. Hold on to that difference; it matters in a few steps.