Effort-moderated (E-M) scoring: excluding rapid-guess responses from test scoring¶

Element · All elements
Evidence · 10 claims (6 for, 1 mixed, 3 against) · 5 studies (2 review, 2 associational, 1 design), q2 · 0 of 5 report an effect size · 10 claims rest on one study

Description¶

Effort-moderated (E-M) scoring is a scoring approach for low-stakes tests in which "item responses classified as rapid guesses are identified and excluded from scoring." It addresses disengaged test taking, which the article describes as "a validity threat that is particularly prevalent with low‐stakes tests." The study evaluated it with achievement test data from quickly retested test takers showing differential disengagement.

Design Implications¶

Context¶

Requirements¶

• Item responses must be classifiable as rapid guesses so they can be identified and excluded from scoring

Constraints¶

• Does not account for all score distortion; nonrapid less-than-full-effort responses remain, per the study's third finding

Target Learners¶

• test takers taking low-stakes achievement tests

Target Learning Goals¶

• valid measurement of achievement despite disengaged test taking

Claims¶

• A modified E-M scoring method using more liberal time thresholds accounts for two-thirds or more of score distortion, outperforming standard thresholds [+W]
• E-M scoring leaves residual score distortion, suggesting some disengaged test responses are nonrapid rather than rapid guesses [-W]
• Standard effort-moderated scoring accounts for roughly one-third of score distortion due to differential disengagement on low-stakes achievement tests [+W]
• Effort-moderated scoring estimates disengaged test takers' performance by excluding disengaged responses and scoring the remaining responses [+W]
• E-M scoring assumes the remaining responses are unaffected by disengagement, and this assumption can fail under partial engagement [-W]
• A method for identifying partial test-taking engagement is validated, with effort-moderated scores to be interpreted cautiously when partial engagement is indicated [~W]
• Test takers generally disengage during only a portion of their items, and these disengaged responses reflect a momentary cessation of measurement [+W]
• Test-taking disengagement distorts both individual and aggregated test scores [+W]
• Responses used in effort-moderated scoring can sometimes reflect less-than-full, or partial, engagement, making E-M scores less trustworthy [-W]
• Disengaged test-taking responses can distort the content representation of test items [+W]

Related Elements¶

• Two techniques for accounting for test disengagement: sample removal and rapid-guess score adjustment

Examples¶

-

Key Sources¶

• Steven Wise, Megan Kuhfeld. (2020). Using retest data to evaluate and improve effort-moderated scoring. Journal of Educational Measurement. https://doi.org/10.1111/jedm.12275