Two factors and their combined effect
Today let's understand what happens when two things are changing at once in the same experiment, and how to tell which one is actually responsible for a result.
A growth team at an online retailer is testing three checkout button designs: Control, Bigger button, and Urgency text. At the same time, every visitor who sees one of those designs also arrives from one of four traffic sources: Organic, Paid search, Email, or Social. Instead of testing the button alone, the team ran all twelve design-source combinations, one week each, and recorded that week's conversion rate, the percent of visitors who completed checkout.
Here are all twelve weeks, plotted together first and then split into one small chart per traffic source.
Flip the toggle. In the combined chart, the three designs sit close together inside every traffic source. Once each source gets its own small chart, Urgency text turns out to sit above the other two designs in every single one of them.
One factor alone says the three designs tie, and why
Let's bring that same data into R and see what a plain test on design alone says, before source enters the picture at all.
Build the data exactly as recorded, then compare the three designs' own average conversion rate against the grand average of all twelve weeks, and against the four sources' own averages too.
"conversion" is each week's percent of visitors who checked out, and "design" has three factor levels: the three button designs under test.
Read the two tapply() lines first. The grand mean, the average of all twelve weeks regardless of design or source, is 3.6. The three designs sit at 3.33, 3.48 and 3.99, a spread of only 0.66 points, all close to each other and close to the grand mean.
The four sources tell a completely different story. They run from 1.6 (Social) up to 6.5 (Email), a spread of 4.9 points, more than seven times design's own spread.
aov(conversion ~ design) fits a one-way test using design alone, and the result, F = 0.102 on 2 and 9 degrees of freedom, p = 0.904, says the three designs look statistically identical. By this test alone, you would call it a toss-up and ship whichever design you liked.
That conclusion is wrong, and the Residuals row shows why. A one-way model only knows about design. Every bit of variation it cannot explain, including the huge swing between sources, gets dumped into that one residual term. Source here is what statisticians call a blocking factor: a factor whose own effect on the outcome is real, often large, but is not the thing the test is actually asking about. Left unnamed, its variation still has to go somewhere, and it inflates the very residual that design's small signal is being measured against.