The misuse of asterisks in hypothesis testing

Open almost any empirical psychology journal and you will find the little clusters of asterisks hanging off test statistics. One star for a result below the 0.05 threshold, two for 0.01, three for 0.001. The convention feels so settled that few people stop to ask whether it actually agrees with the theory it claims to rest on. Dieter Rasch, Klaus D. Kubinger, Jörg Schmidtke and Joachim Häusler argue that it does not. The habit, as they put it, has no footing in the Neyman-Pearson account of statistical hypothesis testing.

The problem starts with when the risk is chosen. In the Neyman-Pearson framework you fix the type-I risk, the chance of wrongly rejecting a true null hypothesis, before you look at the data. That number is part of the design, settled in advance along with the sample size. What the asterisk system does instead is let the researcher read the result first and then announce which threshold it happened to clear. A finding that squeaks past 0.001 gets dressed up as more impressive than one that only reaches 0.05, as if the smaller apparent risk were a property of the evidence rather than a line drawn after the fact.

Rasch and his coauthors are blunt about what that amounts to. Claiming after the fact that a result survives even a strict type-I risk does not prove anything about its strength; it mostly reveals that the researcher is fooling himself. The retroactive relabeling adds no information. It converts a single decision rule into a sliding scale of self-congratulation.

Their second point is more constructive. Much of the temptation to hand out stars comes from working with a fixed sample size, where you collect a predetermined number of observations and then see how far below some cutoff the statistic lands. Switch to a sequential sampling procedure, where observations are gathered one batch at a time and the decision to stop depends on what has accumulated, and the whole asterisk ritual simply has no place to appear. There is a decision to continue or to stop, not a p-value to decorate.

To back the recommendation the paper reports a simulation study comparing the two approaches. Sequential sampling comes out ahead on economy, reaching a decision with fewer observations on average. It also holds up when the data depart from normality, staying robust across non-normal distributions while keeping its power. So the sequential method is not a trade of rigor for convenience; it tends to need less data and tolerate messier data at the same time.

The work appeared in 2004, volume 46, issue 2, from Pabst Science Publishers, with Rasch writing from the BioMath Institute of Applied Mathematical Statistics in Biology and Medicine in Rostock.