Lesson 2 of 2 - comparing groups

Comparing groups with t-tests

In Lesson 1, Meena tested one machine against a fixed target of 1000 g, and you built the whole four-step loop: assume nothing is wrong, compute signal over noise, find the tail area, decide. Here is that loop again - your map for everything below.

But Meena's next question is different. Her old machine is wearing out, and a salesman is offering a new one. She does not have a fixed target to test against any more - she wants to compare two machines against each other: do they fill to different average weights? That is the two-sample t-test, and it carries a twist that can flip the verdict. By the end you will be able to:

  • Run a two-sample and a paired t-test in R, and know which one a situation needs
  • Tell a statistically significant result apart from a large, important one
  • Name the three ways the t-test is most often misused, and how to guard against each

Prerequisites: Lesson 1 (the one-sample t-test: the standard error, the t-statistic, the null distribution, and the p-value).

Comparing two wobbling averages

The two-sample test, and why pairing can flip it

The logic does not change one bit - it is still signal over noise. The only difference is that the signal is now the gap between two averages, and the noise is the standard error of that gap, which has to combine the wobble of both groups:

\[ t = \frac{\bar{x}_1 - \bar{x}_2}{SE_{\text{diff}}}, \qquad SE_{\text{diff}} = \sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}} \]

Read the noise term slowly. Each machine has its own wobble - \(s_1/\sqrt{n_1}\) for the new one, \(s_2/\sqrt{n_2}\) for the old, where \(s_1, s_2\) are the two groups' standard deviations and \(n_1, n_2\) their sizes. Why add the two squared wobbles under the root? Because either average can drift on its own, so the gap between them is shakier than either average alone - the noise of the difference has to carry both machines' wobble at once. After that it is the same move as Lesson 1: the signal (the gap) over the noise (that combined wobble).

To run it on real data, R ships with sleep - results from a genuine 1908 experiment where 10 patients each tried two different sleep medicines, and someone recorded extra, the extra hours of sleep each patient got on each drug. You write outcome ~ group, and by default R runs the Welch version, which does not assume the two groups are equally spread - a safe default.

RInteractive R
# extra hours of sleep, drug 1 vs drug 2 t.test(extra ~ group, data = sleep) #> Welch Two Sample t-test #> t = -1.8608, df = 17.776, p-value = 0.07939 #> mean in group 1 mean in group 2 #> 0.75 2.33

  

Drug 2 gave 1.58 more hours on average, but with \(p = 0.08\) that gap is not quite distinguishable from noise. Now the twist: these were the same 10 patients measured on both drugs - so the two columns are not independent groups, they are paired, patient by patient. (It is like measuring the same patients before and after a treatment - or Meena comparing her machine's output before a service and again after it - so each subject is matched against itself, and the big differences between one subject and the next cancel out.) Telling R to pair them lets each patient cancel out their own personal sleep habits:

RInteractive R
t.test(extra ~ group, data = sleep, paired = TRUE) #> Paired t-test #> t = -4.0621, df = 9, p-value = 0.002833

  

Same data, same averages - but \(p\) drops from 0.08 to 0.003. The effect was real all along; the unpaired test just could not see it through the large differences between one patient and the next. Choosing the right variant is not a formality - it changed the answer.