Does moving items around in a personality questionnaire change what it measures? An IRT analysis of item order

Personality questionnaires draw plenty of criticism, yet they remain one of the standard tools psychologists reach for when they need to assess someone. That tension is the starting point for this study. People keep using these instruments, so the practical question is what happens to them when researchers treat them casually: pulling out individual scales, reordering items, assembling shortened versions without checking whether the measurement properties survive the surgery.

Shifting an item's position sounds harmless. The same question is still there, just earlier or later in the booklet. The point of the study was to test whether that assumption holds up under test theory, and in particular whether classical scoring would even notice a problem if one existed.

Two groups of participants completed the EPP-D (Eysenck, Wilson, & Jackson, 1998). The items used had been selected to be Rasch-homogeneous. One group got the questionnaire in its usual sequence. The other group answered the identical items in exactly reversed order, last item first and so on down the list.

Compared at the level of mean scores, the two versions looked interchangeable. Statistical tests on the averages turned up no significant difference between the conventional order and the reversed one. Someone relying only on that comparison would conclude that order does not matter.

Item response theory told a different story. When the items were examined individually, three of the seven scales showed items whose difficulty parameters had shifted, and the shifts traced back to where the item sat in the sequence. The ordering was doing something to how items functioned, even though it left the group averages untouched.

The takeaway for method: mean-score comparisons from classical test theory are too coarse to catch this kind of effect. If you want to know whether rearranging items has quietly altered a questionnaire, you need the item-level detail that IRT provides. Classical approaches simply do not carry enough information to answer the question.

The wider lesson, that how a personality questionnaire is assembled changes what it measures, holds for every instrument of this type, including the ones written for general readers. The most widely used modern framework is the Big Five, or OCEAN. For a non-technical account of how such a test is built and scored, see this guide to the OCEAN personality test.