Measures Are Not Invariant Across Groups Without Error Variance Homogeneity

Comparing groups, whether they form on their own in the world or get created through an experiment, sits at the heart of how psychology studies people. But any claim about how two groups differ rests on a quieter assumption: that the instrument used to measure the construct of interest behaves the same way in each group. If it doesn't, the comparison loses its footing before the analysis even begins.

This paper takes up a requirement that often gets skipped. When researchers test whether a measure is invariant across groups, they also need to show that the error variances are homogeneous across those groups. The two questions are usually treated as separate. They are not.

Part of the argument turns on what error variance actually is. The common assumption treats it as random noise, the kind of fluctuation that averages out. DeShon argues the opposite. Most of what we call error variance comes from systematic sources that the model never accounts for, influences on the measured responses that are real and patterned rather than random. Calling them random is a convenience, not a description.

Those unmodeled systematic sources do something specific to tests of measurement invariance, and the paper works through that effect for the single indicator case, where one measure stands in for the construct. Moving to multiple indicators does not rescue the situation. Multi-indicator models, often assumed to be a fix, still cannot absorb the problem when the error variances differ from group to group.

The conclusion is blunt. Comparing means, variances, or the correlations among scales across groups carries no clear meaning unless strict metric invariance has been established first. Without that, differences that look substantive may be artifacts of measures that simply aren't working the same way in each group.