E-M scoring leaves residual score distortion, suggesting some disengaged test responses are nonrapid rather than rapid guesses¶

Claim · All claims
Evidence · 1 study · associational r? · q2
Currency · newest study 2020 · checked against its sources 2026-10-09 (automated, 1 of 1 entry) · not reviewed by a person · overdue for a recheck since 2023: tool and model accuracy changes quickly

Subclaims¶

q2 i? The inability of E-M scoring to remove all score distortion suggests additional nonrapid item responses reflect less-than-full engagement by some test takers. → Steven Wise 2020

Evidence¶

Steven Wise 2020¶

Steven Wise, Megan Kuhfeld. (2020). Using retest data to evaluate and improve effort-moderated scoring. Journal of Educational Measurement. https://doi.org/10.1111/jedm.12275

q2 · i? · associational · r?

Third reported finding from the retest-data evaluation: the authors interpret residual distortion as evidence of "nonrapid item responses that reflect less‐than‐full engagement by some test takers." This is the authors' interpretation of the pattern.

"Finally, the inability of E‐M scoring to account for all of the score distortion suggests the additional presence of nonrapid item responses that reflect less‐than‐full engagement by some test takers."

Discussion¶

Related Claims¶

• Disengaged test-taking responses can distort the content representation of test items — related
• Test-taking disengagement distorts both individual and aggregated test scores — related
• Test takers generally disengage during only a portion of their items, and these disengaged responses reflect a momentary cessation of measurement — related
• E-M scoring assumes the remaining responses are unaffected by disengagement, and this assumption can fail under partial engagement — related
• Effort-moderated scoring estimates disengaged test takers' performance by excluding disengaged responses and scoring the remaining responses — related
• Responses used in effort-moderated scoring can sometimes reflect less-than-full, or partial, engagement, making E-M scores less trustworthy — possibly the same claim (merge candidate)
• A modified E-M scoring method using more liberal time thresholds accounts for two-thirds or more of score distortion, outperforming standard thresholds — related
• Rapid-guessing behavior can distort test scores and adversely affect measurement — related
• Standard effort-moderated scoring accounts for roughly one-third of score distortion due to differential disengagement on low-stakes achievement tests — related
• Distortive effects of disengagement on school means were diluted by the high proportion of students exhibiting no non-effortful behavior — related
• In simulation, effort-moderated scoring (EM-CAT and EG-CAT) yields negligible bias and nominal 95% confidence interval coverage, unlike MLE-CAT scoring — related
• E-M scoring partially accounts for observed RIT score differences between disengaged first tests and engaged retests — related
• Current measurement models underlying achievement testing assume students respond effortfully to test items — a broader claim this one bears on
• Rapid guesses on computer-based tests reflect a construct-irrelevant response process rather than the tested knowledge, skills, and abilities — related
• Rapid guessing differs from solution behavior as a form of test-taker disengagement — related
• Rapid-guessing behavior on an adaptive achievement test is inconsistent with most proposed models of unmotivated test taking except the effort-moderated model — related
• Very short response times (rapid guessing) indicate disengaged test taking in both high-stakes and low-stakes testing contexts — related
• Technology-enhanced assessment items were studied as a potential means of increasing student test engagement and reducing disengagement — related
• Design effects from aggregation are tolerable with moderate numbers of groups and few covariates — related
• Disengaged student responses raise the issue of whether engaged responses maintain the intended content representation — related
• Rapid guesses on MAP Growth reflect a lack of engagement rather than a lack of knowledge — related
• Seriously disengaged initial testers who stopped rapid-guessing on retest showed large RIT gains, averaging about 6 points in Reading and 12 in Math — related
• Effort-moderated scoring yields more accurate scores than traditional scoring on an adaptive achievement test — related
• A proposed effort-guided adaptive test alleviates item difficulty mistargeting caused by unmotivated test taking, as shown by simulation — related
• Differing response styles create a construct-irrelevant source of bias that distorts fundamental inferences made from survey data — related
• Rapid guessing is a validated indicator of disengaged item responding — related