The robustness of parametric statistical methods

Walk through enough psychology papers and you notice a pattern: researchers reach for non-parametric tests in situations where the parametric version would actually serve them better. The reasoning usually traces back to worry about one assumption. The parametric methods assume normality, since that assumption is what lets you work out the exact distribution of the test statistic, and people treat any possible departure from a normal curve as grounds for abandoning the whole approach.

Consider the t-test and what tends to be offered as its substitute, the Wilcoxon test (also called the Mann-Whitney test). The two do not answer the same question. Wilcoxon compares the full distributions of the two samples. That means it can return a significant result even when the two means are identical, because differences in higher moments between the populations are enough to trigger it. The t-test, by contrast, holds up remarkably well when the data are not normal. It is robust enough that in practice there is almost never a reason to switch to Wilcoxon on grounds of non-normality alone.

This paper collects results from a program of work that looked, in a systematic way, at how statistical procedures behave when the normality assumption fails. The work came out of a research group in Dummerstorf, near Rostock, and was carried out some years back. Until now the findings had not been brought together and published as a single body of results.

The bulk of the evidence rests on simulation. Each study ran 10,000 trials, which gives a large base for judging how estimates and tests perform under various non-normal conditions. Alongside the simulations there are exact results worked out mathematically for the most extreme departures from normality you can construct, namely two-point and three-point distributions, where the shape is as far from a bell curve as it gets.

What comes out of all this is consistent. For inference about means, the parametric approach stays reliable across the great majority of realistic cases. The practical conclusion is that it can be recommended for nearly every application, rather than being set aside at the first sign that the data might not be perfectly normal.