Response likelihood as a measure of the reliability of the response actually given, computed from the 2PL IRT model¶

Theory · All theories
Evidence · no claims cited

Description¶

The article defines response likelihood L = π(θ)^s · (1−π(θ))^(1−s), where π is the 2PL IRT probability that an examinee of ability θ answers correctly and s is 1 for correct or 0 for incorrect. Unlike response accuracy, which ignores the actual response, "The response likelihood is a measure of reliability test taker's response for a given response model": low-likelihood responses are unlikely and may reflect low effort or aberrant behavior such as guessing. The framework rests on the assumption that all test takers use the same responding strategy for an item.

Design Implications¶

Context¶

Requirements¶

• Item discrimination and difficulty parameters from 2PL IRT analysis and an estimate of each test taker's ability (obtained here by maximum likelihood) are needed to compute π and hence likelihood.

Constraints¶

• Likelihood as a reliability measure is based on the assumption that all test takers use the same strategy for responding to an item; if strategies differ, the response time–likelihood relationship depends greatly on the applied strategies.

Target Learners¶

• test takers in computer-based assessment, including low-motivation examinees on low-stakes tests

Target Learning Objectives¶

• identifying unreliable or aberrant item responses to improve ability and item-parameter estimation

Claims¶

Related Theories¶

Examples¶

• Psychometric pipeline for mapping LLM performance onto interpretable proficiency levels and subskill mastery
• Semi-supervised deep learning bias-adjustment framework for IRT parameter estimation under nonignorable missingness

Key Sources¶

• Srdjan Verbić & Boris Tomić. (2008). Test item response time and the response likelihood. Institute for Education Quality and Evaluation. http://sepp.ceo.edu.rs/moodle/mod/resource/view.php?id=25