The sampling distribution of a statistic
Today let's find out whether the mean is the only statistic that settles into a predictable spread, or whether a share, a median and a variance do too.
Kiran runs a small online bookstore. Orders come in all day, some for a single paperback, some for a stack of books at once, so the amounts land all over the place. Here is one day of 30 orders, in dollars, in the order they arrived.
Most of those orders sit well under the dashed line, Kiran's own $35 average order value, but a few tower over it, one even past $150. That is a right skewed shape, the kind with a long thin tail stretching off to the right.
The mean of a sample like this settles into a predictable, bell shaped spread as you average more and more days like it. Whether a share of big orders, a median order and a variance of orders, each built from a chart just like the one above, settle just as predictably is the question, checked one statistic at a time.
The standard error is the spread of the statistic, not of the data
Kiran's bookstore models its daily order values as exponential with a rate of 1/35. That single rate pins down two numbers at once: the population's mean order value, $35, and for this particular distribution, its standard deviation too, also $35.
Draw the spread of Kiran's 30 orders, then draw the spread of their mean once you repeat the sampling many times, and set the two side by side.
orders_day holds that one day's 30 orders, and sd() on them comes out to 37.37, close to the population's own standard deviation of $35. That is the data's spread: how far a single order tends to sit from the average order.
means30 is a different thing entirely. replicate(2000, mean(rexp(30, rate = 1/35))) draws a fresh 30 orders, averages them, and repeats that 2000 times, so it holds 2000 separate daily averages, one per simulated day. Its spread, 6.42, is nowhere near 37.37, because an average of 30 orders moves around far less than a single order does.
This spread of the mean itself, not of the raw data behind it, has a name: the standard error. For the mean, it follows a simple formula, the population's standard deviation over the square root of the sample size. 35 divided by the square root of 30 comes out to 6.39, matching the simulated 6.42 closely.