Reading an Experiment
In Lesson 3 you sized Riverside Books' experiment, ran it, and got a p-value of about \(1.2 \times 10^{-5}\): the "Readers also bought" panel almost certainly does something. That is where most people stop. But a p-value is a yes/no answer, and shipping a product change is not a yes/no decision. Before Riverside rolls the panel out to every visitor, four harder questions remain.
- How big is the effect? A "significant" lift could be one buyer in a thousand or one in twenty.
- How sure are we of that size? A single point estimate hides how much wiggle room the data leaves.
- Does it matter in practice, not just on paper?
- What can we actually claim from one experiment, and what would be overreaching?
This lesson turns a bare "it's significant" into an honest, defensible conclusion. The curve below is that same significance test from Lesson 3: drag the observed result and the p-value hands back its one and only verdict, reject or don't. That single yes/no is everything a p-value can tell you. By the end you will read far more out of the same experiment.
By the end of this lesson you will be able to:
- Compute the absolute effect (in percentage points) and the relative lift (in percent), and explain why the relative number alone can mislead
- Read a 95% confidence interval as a range of plausible true effects, compute it in R, and connect "the interval excludes zero" to "p is below 0.05"
- State correctly what "95% confidence" means, and avoid the reading almost everyone gets wrong
- Separate statistical significance from practical significance, and scope what a single experiment does and does not let you claim
Prerequisites: you finished Lesson 3 (randomization, the null hypothesis, p-values, the significance level \(\alpha\), power, and the minimum detectable effect). Every new term here is defined as it appears.
Effect size: absolute versus relative
The first thing to pull out of a result is its effect size: not "is there a difference" but "how large is the difference." For Riverside's two conversion rates there are two honest ways to say it, and they sound wildly different.
The absolute effect is the plain difference in rates, the extra share of visitors who buy under the new panel:
\[ \Delta = \hat p_B - \hat p_A, \]
where \(\hat p_A\) is the observed control rate and \(\hat p_B\) the observed treatment rate (the hat means "measured from the sample"). Riverside's is about 1.6 percentage points.
The relative lift divides that same gap by the baseline, so it reads as a percent increase over where you started:
\[ \text{relative lift} = \frac{\hat p_B - \hat p_A}{\hat p_A}. \]
Riverside's is about 42%. Same experiment, same 260-versus-368 buyers, described two ways: "1.6 points more" and "42% more buyers." Both are true. Build the counts inline and see them fall out.