Standard effort-moderated scoring accounts for roughly one-third of score distortion due to differential disengagement on low-stakes achievement tests¶

Claim · All claims
Evidence · 1 study · associational r? · q2
Currency · newest study 2020 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? In a retest-based evaluation, standard E-M scoring accounted for roughly one-third of the score distortion attributable to differential disengagement. → Steven Wise 2020

Evidence¶

Steven Wise 2020¶

Steven Wise, Megan Kuhfeld. (2020). Using retest data to evaluate and improve effort-moderated scoring. Journal of Educational Measurement. https://doi.org/10.1111/jedm.12275

q2 · i? · associational · r?

Evaluation using achievement test data from test takers who were quickly retested and showed differential degrees of disengagement; the article reports that "standard E‐M scoring accounted for roughly one‐third of the score distortion due to differential disengagement." No effect size is printed.

"Using achievement test data composed of test takers who were quickly retested and showed differential degrees of disengagement, three basic findings emerged. First, standard E‐M scoring accounted for roughly one‐third of the score distortion due to differential disengagement."

Discussion¶

Related Claims¶

• Test-taking disengagement distorts both individual and aggregated test scores — related
• Effort-moderated scoring estimates disengaged test takers' performance by excluding disengaged responses and scoring the remaining responses — related
• Responses used in effort-moderated scoring can sometimes reflect less-than-full, or partial, engagement, making E-M scores less trustworthy — related
• A modified E-M scoring method using more liberal time thresholds accounts for two-thirds or more of score distortion, outperforming standard thresholds — related
• E-M scoring leaves residual score distortion, suggesting some disengaged test responses are nonrapid rather than rapid guesses — related
• In simulation, effort-moderated scoring (EM-CAT and EG-CAT) yields negligible bias and nominal 95% confidence interval coverage, unlike MLE-CAT scoring — related
• E-M scoring partially accounts for observed RIT score differences between disengaged first tests and engaged retests — possibly the same claim (merge candidate)
• Seriously disengaged initial testers who stopped rapid-guessing on retest showed large RIT gains, averaging about 6 points in Reading and 12 in Math — related