Testing on ranks with Wilcoxon and Kruskal-Wallis

Today let's understand two tests that work on the rank of a value instead of the value itself: the Wilcoxon rank-sum test and the Kruskal-Wallis test.

A company is comparing two designs for its checkout page, A and B. Design A was timed on 5 customers, in seconds: 38, 45, 52, 60 and 150. Design B was timed on 6 different customers: 30, 33, 41, 44, 48 and 55.

Design A's checkout times average 69 seconds. Design B's average 41.83 seconds, a 65% gap that makes A look much slower to check out with. But that gap is driven almost entirely by one customer on design A who took 150 seconds. Look at the middle value instead: design A's median is 52 seconds, design B's is 42.5, far closer together.

The chart below plots every one of those 11 checkout times, grouped by design.

On the boxplot above, design A's box and whisker stretch much higher than design B's, pulled up by that single 150-second point sitting on its own near the top.

One outlier, two very different averages

Start with the two numbers that make design A look so much slower: a mean of 69 seconds against design B's 41.83.

Compute all four numbers in R to see exactly how far apart they sit.

RInteractive R
# Compare the mean and median checkout time for design A and design B A <- c(38, 45, 52, 60, 150) B <- c(30, 33, 41, 44, 48, 55) mean(A) #> [1] 69 mean(B) #> [1] 41.83333 median(A) #> [1] 52 median(B) #> [1] 42.5

  

A mean adds up every value and divides by how many there are, so one unusually large value pulls it a long way. A median only looks at the middle value once the data are sorted, so it barely notices how large the largest value actually is, only that it is the largest.

See how much of that mean comes from the single slowest checkout by dropping it and recomputing.

RInteractive R
# Recompute design A's mean with the 150-second checkout removed mean(A[A != 150]) #> [1] 48.75

  

Removing one customer out of five drags design A's mean down from 69 to 48.75, a large swing for a single dropped value. Its median, 52, would barely move if that same customer dropped out, because the middle of five sorted values rarely depends on which value sits at the very end.

That is exactly the property the Wilcoxon rank-sum test is built around: a statistic that, like the median, pays attention to where the 150 sits in the order of all the checkout times, never to how large it actually is.