Reporting effect size, not just significance

Today let's understand effect size: the question a p-value never answers, which is how big a result actually is.

Here is the pilot we will use throughout. A school tries out a weekly tutoring program. 12 students get the tutoring, another 12 get the standard program instead, and after a term all 24 sit the same 50-point test. The tutored group averages 36.5. The standard group averages 31.17. A t-test run on these 24 scores gives t = 2.22, p = 0.037, so the gap clears the usual 0.05 bar.

But p = 0.037 only says the gap is unlikely to be pure noise. It says nothing about whether 5.33 points, out of 50, is a result worth changing a program for. That is a separate question, and it is the one this lesson answers.

The chart below shows the 24 scores split by group. Switch between the boxplot and the point view to see the same gap two ways.

The raw difference, in the test's own units

Start with the plainest number there is: how far apart are the two group averages, in the units the test was actually scored in.

RInteractive R
# Compute each group's mean score and the raw gap between them tutored <- c(38, 37, 30, 34, 40, 40, 31, 36, 28, 36, 45, 43) standard <- c(32, 39, 37, 34, 27, 32, 39, 36, 29, 20, 29, 20) group_means <- tapply(c(tutored, standard), rep(c("tutored", "standard"), each = 12), mean) group_means round(group_means["tutored"] - group_means["standard"], 2) #> standard tutored #> 31.16667 36.50000 #> tutored #> 5.33

  

The tutored group scores 5.33 points higher on average. That is the raw difference, and it is already a complete, meaningful number: a teacher reading this result does not need anything standardized to understand it. 5.33 points, out of 50, is just 5.33 points.

But the raw difference has a limit. It is tied to this test's own scale. If the test had been out of 100 instead of 50, the same underlying improvement could show up as twice the gap, a bigger-looking number for the exact same effect. So a raw difference from one test cannot be compared to a raw difference from a different test, or a different outcome, unless both are measured on the same scale.

That is the gap the next few numbers are built to close.