Checking assumptions and what breaks them
Today let's understand why a statistical test's assumptions are not all equally risky to break.
A school district wants to know if more tutoring helps. It picks 12 classrooms of 5 students each, 60 students in total, and gives each classroom a different number of weekly tutoring hours, from 0 up to 10. At the end of term it records every student's gain on a final exam, then fits a regression line through the 60 points: tutoring hours on one axis, exam gain on the other.
That line has a slope, and the slope comes with a standard error and a 95% confidence interval. The test behind that interval is the same t machinery behind an ordinary t-test, and it rests on three assumptions: the errors around the line are normally distributed, they spread out by the same amount regardless of tutoring hours, and every student's result is independent of every other student's.
Those three are not equally dangerous to break, as the labels above already hint: one barely registers, one does real damage, and one sits in between.
The classroom tutoring study, and the test riding on it
Let's build that tutoring study for real, so every number in this lesson has somewhere to come from.
Twelve classrooms of five students each get a different number of weekly tutoring hours: classroom 1 gets 0 hours, classroom 12 gets 10 hours, and the ten classrooms in between get hours spread evenly across that range. Every student's exam gain is the classroom's tutoring hours times a fixed effect, plus ordinary random noise.
Press Run.
Read the first line. Every extra hour of tutoring raises exam gain by about 1.6 points, with a standard error of 0.19. The t value is 8.47 and the p-value is far below the usual 0.05 cutoff, small enough to round to 0.000 at three decimals.
The second line is the 95% confidence interval for that slope: 1.22 to 1.98. That interval, and the test behind it, is the real subject here: whether it still means what it claims once the assumptions under it are broken. The third line, R-squared of 0.553, says this one line explains about 55% of the variation in exam gain. Hold onto that number: a later step shows R-squared staying almost exactly this calm even while the interval above it falls apart.
Every one of those three numbers, the slope's standard error, the confidence interval, and R-squared, is computed assuming the errors around the line are normally distributed, equally spread out, and independent of each other.