Identifying rater effects using latent trait models

Human judges disagree. When people score essays or rate performances, their scores carry more than just the quality of the work in front of them; they carry habits, biases, and quirks that belong to the rater rather than the examinee. This paper looks at how latent trait models can pull those rater-specific patterns out of scoring data so they can be measured directly.

The particular tool here is the multi-faceted Rasch model. Rather than treating a score as a simple reflection of essay quality, the model separates out the different sources that feed into it, including the person being rated and the rater doing the rating. That separation is what makes it possible to put a number on how an individual rater behaves.

Three broad categories of rater effect are laid out. The first concerns accuracy, or the lack of it: whether a rater's judgments track the actual quality of the work or scatter away from it. The second is severity versus leniency, the familiar problem of the tough grader and the easy one, where two raters looking at the same essay land in different places because of where they personally set the bar. The third is centrality versus extremism, the tendency either to cluster scores around the middle of the scale and avoid the ends, or to reach for the extremes more readily than the evidence warrants.

For each of these effects the paper identifies indices, statistical indicators derived from the model that flag when a rater is drifting in one of these directions. The point is to have something concrete to look at rather than a vague suspicion that a particular reader is grading oddly.

The indices are then put to work on real data: essay scores from an Advanced Placement examination. These were not casual judgments. The raters involved had gone through extensive training, which matters, because it means the effects being hunted for are not simply the artifacts of untrained or careless scoring.

Even among those well-prepared readers, the effects showed up. Specific raters suspected of inaccuracy, of excess severity or leniency, and of central or extreme tendencies could be singled out from the data. And the problem was not trivial in size. The effects appeared at rates and magnitudes that cannot responsibly be set aside, which is to say that trained human scoring still leaves measurable rater influence in the results.