Responses used in effort-moderated scoring can sometimes reflect less-than-full, or partial, engagement, making E-M scores less trustworthy¶

Claim · All claims
Evidence · 1 study · design r? · q2
Currency · newest study 2021 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · overdue for a recheck since 2024: tool and model accuracy changes quickly

Subclaims¶

q2 i? A prior study provided evidence that responses retained in E-M scoring sometimes reflect partial engagement, so E-M scores can still be distorted by disengagement. → Steven Wise 2021

Evidence¶

Steven Wise 2021¶

Steven Wise, Megan Kuhfeld. (2021). A method for identifying partial test-taking engagement. Applied Measurement in Education, 34(2), 150-161. https://www.nwea.org/research/publication/a-method-for-identifying-partial-test-taking-engagement/

q2 · i? · design · r?

The article reports, citing a recent study, that responses used in E-M scoring "can sometimes reflect less-than-full, or partial engagement", and that in these instances E-M scores "can still be distorted by disengagement". No statistics are printed in the available text.

"A recent study provided evidence that the responses used in E-M scoring can sometimes reflect less-than-full, or partial engagement. In these instances, E-M scores will be less trustworthy because they can still be distorted by disengagement."

Discussion¶

Related Claims¶

• Disengaged test-taking responses can distort the content representation of test items — related
• Test-taking disengagement distorts both individual and aggregated test scores — related
• Test takers generally disengage during only a portion of their items, and these disengaged responses reflect a momentary cessation of measurement — related
• E-M scoring assumes the remaining responses are unaffected by disengagement, and this assumption can fail under partial engagement — possibly the same claim (merge candidate)
• E-M scoring leaves residual score distortion, suggesting some disengaged test responses are nonrapid rather than rapid guesses — possibly the same claim (merge candidate)
• A method for identifying partial test-taking engagement is validated, with effort-moderated scores to be interpreted cautiously when partial engagement is indicated — related
• Standard effort-moderated scoring accounts for roughly one-third of score distortion due to differential disengagement on low-stakes achievement tests — related
• The authors provide a method for identifying partial test-taking engagement, supported by validation evidence — related
• Distortive effects of disengagement on school means were diluted by the high proportion of students exhibiting no non-effortful behavior — related
• Current measurement models underlying achievement testing assume students respond effortfully to test items — a broader claim this one bears on
• Very short response times (rapid guessing) indicate disengaged test taking in both high-stakes and low-stakes testing contexts — related
• RTE provides an assessment of test-taking effort down to the level of individual item responses — related
• Design effects from aggregation are tolerable with moderate numbers of groups and few covariates — related
• Disengaged student responses raise the issue of whether engaged responses maintain the intended content representation — related
• Technology-enhanced assessment items were studied as a potential means of increasing student test engagement and reducing disengagement — related
• A modified E-M scoring method using more liberal time thresholds accounts for two-thirds or more of score distortion, outperforming standard thresholds — related