Comparing Tests for 2 x 2 Tables and Two-Sample Configural Frequency Analysis

When you work with a 2 x 2 table, you have a surprising number of statistical tests to choose from, and they do not all behave the same way. This piece reports on a simulation study that put five of them side by side. Each one can be applied to 2 x 2 tables generally, and each is relevant to two-sample Configural Frequency Analysis (CFA) specifically. The five are Pearson's X2 test, two different z-approximations, the log-odds ratio test, and Fisher's exact test.

The first thing the simulations tell us concerns relative power. Ranked from most to least powerful, the ordering came out as: the first z-approximation, then Fisher's exact test, then Pearson's X2, then the second z-approximation, with the log-odds ratio test trailing at the bottom. So the choice of test is not cosmetic. It changes how readily you detect a real effect.

Power is only part of the story, though. A test can be powerful and still misbehave in the sense that its statistic does not follow the sampling distribution you assume when you read off a p-value. Among the four approximative tests studied here, two produced statistics that tracked their respective sampling distributions reasonably well. The other two did not. The first z-approximation was one of the offenders, and the log-odds ratio test statistic was the other.

A separate distinction showed up clearly in the results: some of these tests are marginal-free and some are marginal-dependent. The simulation made that property visible rather than leaving it as a theoretical footnote, which matters when you are deciding what the margins of your table should be allowed to do.

Putting the pieces together, the practical recommendation for two-sample CFA is to lean on Pearson's X2 test and the second z-test. That holds with particular force in situations where Fisher's exact test is not available to you.

Alexander von Eye, Department of Psychology, Michigan State University.