The Robustness of Two-Sample Tests for Means: A Reply to von Eye's Comment

When you need to compare the means of two distributions, which test deserves your trust: the t-test or the Wilcoxon test? Our answer comes down on the side of the t-test, and the reasoning has less to do with any single property than with how each test holds up once the textbook assumptions start to slip.

Start with the assumption people worry about most, normality. Neither test is especially fragile here. Both tolerate departures from a normal distribution reasonably well, and if anything the Wilcoxon test carries a slight edge in power under those conditions. That edge is real, but it is narrow, and it is only one part of the picture.

The picture changes once the two samples differ in variance. The t-test absorbs that kind of heterogeneity without much trouble. The Wilcoxon test does not. A similar gap shows up when the underlying distribution is discrete rather than continuous: the t-test keeps behaving sensibly, while the Wilcoxon test loses its footing. So the Wilcoxon test's small advantage in power buys you fragility in two settings that come up often in practice.

There is one failure both tests share, and it is worth being blunt about it. If the observations inside a sample are not independent, if they carry autocorrelation or some other form of dependence, robustness collapses for both procedures. No clever choice between the two rescues you from that.

The remedy for dependence is not statistical, it is practical. Design the survey or experiment so that observations within a sample do not lean on one another in the first place. Get that right, and the comparison between the t-test and the Wilcoxon test reduces to the trade-off above, where the t-test's steadiness across unequal variances and discrete data makes it the safer default.