Adaptive Success Control in Computerized Adaptive Testing

Computerized adaptive testing, or CAT, works inside the framework of probabilistic test theory by tuning the difficulty of each item to the ability of the person answering it. The goal is to squeeze the most information out of every single item. Do that well and you gain on two fronts: the test becomes more economical, and it feels more reasonable to the person taking it.

There is a catch that earlier studies have pointed to. When an item is calibrated so the respondent has roughly a fifty-fifty chance of answering correctly (a success probability of p=0.5), some people feel pushed too hard. They struggle to reach a level of certainty they are comfortable with inside a sensible amount of time. So they slow down, and the time spent on each item climbs.

Depending on what the test is actually measuring, that extra time can eat up the advantage of serving perfectly informative items. The irony is that CAT, meant to save effort, can end up costing it. Instead of a leaner test you get a longer one, and the economy you were chasing slips away.

This is the problem Joachim Häusler set out to address by building and trying an adaptive success control algorithm. Rather than holding the success probability fixed, the method adjusts it to suit how a given person tends to work. Respondents who need more certainty before they will commit to an answer get flagged, and the algorithm hands them items with a higher success probability. The point is to shorten how long the test takes, which is a different target from classical CAT. Classical CAT tries to cut the number of items; this approach tries to cut duration.

To check whether it holds up, the algorithm was run against data from the Adaptive Matrices Test (AMT), developed by Hornke, Etzel and Rettig in 1999. Häusler re-analyzed that data and then compared two versions of the AMT head to head: one using standard CAT and an experimental one using Adaptive Success Control.

The findings are weighed against both the psychometric and the psychological sides of what makes a test good. Keywords for the work: CAT, adaptive testing, test economy, ASC, success control. The author, Mag. Joachim Häusler, was at Dr. G. Schuhfried GmbH in Mödling, Austria.