Bias, variance, and the cost of an estimator
Today let's work out what actually makes one formula for estimating a number better than a rival formula, not by guessing, but by scoring both against a truth we happen to know.
A quality engineer on a bolt-manufacturing line pulls n = 6 bolts off the line every shift and measures their diameters, in millimeters. When the line is correctly calibrated, those diameters come from a process with a true mean of 10 mm and a true variance of 0.04 square millimeters, a standard deviation of 0.2 mm.
She wants a formula she can trust to estimate that variance from just one shift's six bolts. Two formulas compete for the job. One divides the sum of squared deviations from the sample mean by n minus 1, which is 5, exactly what R's var() function returns. The other divides that same sum by n, which is 6, the maximum likelihood estimator.
The widget above shows one way a set of repeated estimates can go wrong: a cluster of shots sitting off to one side of the bullseye, but packed in close together.
Picturing bias and variance on a target
Picture an estimator fired at a true value over and over, once per sample it is handed. Bias is how far the average of all those shots sits from the true value. Variance is how spread out the individual shots are around that average, regardless of where the average lands.
Drag the bias slider down to 0 and push the variance slider up past 0.7. The shots spread out widely now, but their average still centers on the bullseye: zero bias, but a lot of variance.
Now push bias back up past 1, and bring variance back down below 0.7. The shots pull back into a tight little cluster, but that cluster sits off away from the bullseye again: bias is back, and variance has dropped.
Bias and variance are two separate properties of how an estimator behaves across repeats. An estimator can carry either one without the other, or some of both, and nothing about one tells you the value of the other. This dartboard is not using the bolt line's real numbers. It is simply a way to feel what bias and variance mean on their own, independent of any one particular estimator.
Bias and variance for two competing formulas
Testing one shift's six bolts only gives one estimate from each formula, and one estimate is not enough to say which formula is better. The real test is to repeat the shift many times, apply both formulas every time, and see how their estimates behave across all those repeats.
Simulate 50000 shifts of six bolts each, all drawn from the true process, mean 10 mm and variance 0.04 square millimeters, and score both formulas against that true variance. A real quality engineer never gets to see the true variance directly, but this simulation does, which is exactly why it can settle the question.
Across the 50000 simulated shifts, the first formula, divide by 5, averages 0.040111. That average sits just 0.000111 above the true variance, 0.04, a bias close enough to zero to call this formula unbiased. Its variance across those 50000 estimates is 0.000649, a measure of how much any one shift's estimate wanders from that average.
The second formula, divide by 6, averages 0.033426, pulled down to 0.006574 below the true variance. That is a real, measurable bias, not noise from the simulation. But its variance is only 0.000451, tighter than the first formula's 0.000649.
If you ever compute var() on a single sample in R, that is exactly the first formula at work, the sum of squared deviations divided by n minus 1. R's default variance estimator is unbiased for exactly this reason.
Quick check: reading bias and variance apart
Of the bolt line's two formulas just computed, divide by 5 averaged 0.040111 (bias 0.000111, variance 0.000649) and divide by 6 averaged 0.033426 (bias -0.006574, variance 0.000451). Which one is the estimator sitting almost exactly on the true variance, and which one is the estimator whose estimates cluster more tightly around their own, slightly wrong, average?
Mean squared error: combining bias and variance into one number
Bias and variance score an estimator on two separate scales, and neither formula is a clean winner so far: divide by 5 wins on bias, divide by 6 wins on variance. To actually compare them, both scores need to combine into one number. That number is mean squared error, MSE, and it is just bias squared plus variance.
MSE also has a second, more direct definition: the average, across all 50000 simulated estimates, of the squared gap between that estimate and the true variance. The two definitions should agree, since they measure the exact same thing two different ways, and the code below checks that they do.
Both routes land on the same two numbers. Divide by 5's MSE comes to 0.0006489, whether you add its bias squared to its variance or average the squared gap directly. Divide by 6's MSE comes to 0.0004938, by either route.
Despite carrying a real bias, divide by 6 has the lower MSE, about 24% below divide by 5's. The formula most statistics courses teach you to prefer is not the one that scores best here.
Why a biased estimator can have lower MSE: the arithmetic
Line the four numbers up for each formula and the trade becomes arithmetic, not a trick.
| Formula | Bias | Bias squared | Variance | MSE |
|---|---|---|---|---|
| Divide by 5 | 0.000111 | effectively zero | 0.000649 | 0.000649 |
| Divide by 6 | -0.006574 | 0.0000432 | 0.000451 | 0.000494 |
Dividing by 6 instead of 5 cuts variance from 0.000649 to 0.000451, a drop of 0.000198, about 31% less. That same change in divisor pulls the average estimate further from the truth, so bias grows in size from 0.000111 to 0.006574. But squaring that bias before it enters MSE shrinks it enormously: 0.006574 squared is only 0.0000432, a little over a fifth of the 0.000198 of variance it bought.
MSE is just bias squared plus variance, added up the same way whether an estimator is biased or not. When a small bias buys a much bigger cut in variance, as it does here, the biased estimator's MSE ends up lower. Nothing protects an unbiased estimator from losing this trade. MSE counts exactly what each estimator actually costs, and here the unbiased one costs more.
Sliding the divisor: the full bias-variance curve
Divide by 5 and divide by 6 are not the only two choices. Nothing stops you from dividing that same sum of squared deviations by any other number, so sweep the divisor from 4 to 10 and score bias, variance and MSE at every one, using the same 50000 simulated shifts already in hand.
Read down the bias column and it only gets more negative: a bigger divisor always pulls the estimate down further below the truth. Read down the variance column and it only gets smaller: a bigger divisor always shrinks the spread. Those two columns move in opposite directions the whole way down.
MSE, the last column, does neither. It falls from 0.001117 at divisor 4, down to a low of 0.000460 at divisor 7, then climbs back up to 0.000560 at divisor 10. Divisor 7 is n plus 1, one more than even the maximum likelihood estimator's divisor of 6, and its MSE beats both of the named formulas from earlier, even though its bias, -0.011349, is larger in size than either named formula's own bias.
The same U shape shows up whenever an error trades off two pieces that move in opposite directions. The widget below is not using the bolt line's own numbers. It plots a different, more familiar version of the same trade: a model's error on data it has not seen, as the model is allowed to get more flexible.
Step its slider across and watch the same shape: one error falling as the model fits the data it has already seen better and better, the other error falling for a while and then climbing back up, the same way the MSE column above fell and then climbed. The slider's exact numbers belong to that different example, not to the bolt line, but the shape it draws is the one divisor 4 through 10 just traced for the bolt line's own variance.
Your turn: score the divisor with the lowest MSE
ss still holds every simulated shift's sum of squared deviations. Divide it by 7 instead of 5 or 6, the way the sweep above did, then score that estimator's bias, variance and MSE exactly the way divide by 5 and divide by 6 were scored earlier.
Show answer
# Divide each sample's sum of squared deviations by 7 and score the result the same way
s2_d7 <- ss / 7
bias_d7 <- mean(s2_d7) - true_var
variance_d7 <- var(s2_d7)
mse_d7 <- bias_d7^2 + variance_d7
round(c(bias = bias_d7, variance = variance_d7, mse = mse_d7), 6)
#> bias variance mse
#> -0.011349 0.000331 0.000460 Quick check: what MSE actually measures
Here is a different, unrelated estimation problem. Estimator P is unbiased: bias 0, variance 0.12. Estimator Q is biased: bias 0.25, variance 0.05. Which one has the lower mean squared error?
References
- Casella, G. and Berger, R. L., Statistical Inference (2nd ed., Duxbury/Chapman and Hall), bias, variance and mean squared error for competing estimators of a normal variance
- Rice, J. A. (2007). Mathematical Statistics and Data Analysis (3rd ed.). Duxbury. The worked comparison of variance estimators by mean squared error.
- Wasserman, L., All of Statistics (Springer, 2004), the chapter on point estimation and the bias-variance-MSE framework used throughout this lesson
- Wikipedia, "Bias of an estimator", the minimum-MSE divisor result for estimating a normal variance.
- Wikipedia, "Mean squared error", the bias-variance decomposition of MSE.
What this trade-off means for every estimator you build next
The formula most people are taught to reach for by default, divide by n minus 1, was not the one with the lowest MSE on the bolt line's own variance. Divide by 6 beat it, and divide by 7 beat them both, trading a bigger bias for an even bigger cut in variance.
Picture the dartboard once more. A cluster of shots sitting off the bullseye but packed in tight is bias without much variance, and a cluster centered on the bullseye but scattered wide is variance without much bias. Divide by 5's error on the bolt line was almost entirely variance, 0.000649 of it against a bias squared too small to matter. Divide by 6's error was 0.0000432 of bias squared sitting alongside 0.000451 of variance, a worse-looking dart but a better-scoring one.
Being unbiased is not the same as being good. Mean squared error is the number that actually settles which estimator wins, and it counts bias and variance on exactly equal terms.