Test-taking disengagement distorts both individual and aggregated test scores¶

Claim · All claims
Evidence · 1 study · review r? · q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2024: tool and model accuracy changes quickly

Subclaims¶

q2 i? Disengaged test taking has a distortive impact on both individual scores and aggregated scores. → Steven Wise 2021

Evidence¶

Steven Wise 2021¶

Steven Wise. (2021). Six insights regarding test-taking disengagement. Educational Research and Evaluation, 26(5-6), 328-338. https://www.nwea.org/research/publication/six-insights-regarding-test-taking-disengagement/

q2 · i? · review · r?

The article states accumulated research has advanced "our understanding of its distortive impact on both individual and aggregated scores". This is the author's synthesis of prior research, reported at the review level without effect sizes.

"This issue has received a growing research interest over the past 20 years, with notable advances in both the measurement of disengagement as well as our understanding of its distortive impact on both individual and aggregated scores."

Discussion¶

Related Claims¶

• Disengaged test-taking responses can distort the content representation of test items — related
• Two decades of research have produced advances in both the measurement of test-taking disengagement and understanding of its distortive impact on individual and aggregated scores — possibly the same claim (merge candidate)
• Rapid-guessing behavior can distort test scores and adversely affect measurement — related
• Disengaged test taking materially impacts individual state summative test scores but may have relatively minor impact on score aggregations — reports the opposite
• E-M scoring leaves residual score distortion, suggesting some disengaged test responses are nonrapid rather than rapid guesses — related
• Standard effort-moderated scoring accounts for roughly one-third of score distortion due to differential disengagement on low-stakes achievement tests — related
• Responses used in effort-moderated scoring can sometimes reflect less-than-full, or partial, engagement, making E-M scores less trustworthy — related
• School officials regularly use school-aggregate test scores to monitor school performance and make policy decisions — related
• Twenty years of research on disengaged test taking in international assessment programs yields six insights with implications for assessment programs — related
• Disengagement produced both positive and negative distortion of individual student scores, which tended to cancel out much of the net distortive effect on school means — a narrower finding that bears on this claim
• Very short response times (rapid guessing) indicate disengaged test taking in both high-stakes and low-stakes testing contexts — related
• RTE provides an assessment of test-taking effort down to the level of individual item responses — related
• School-level aggregated scores on the computer-based PISA-Based Test for Schools were highly stable despite considerable differential engagement among schools — a narrower finding that bears on this claim
• Students scored an average of 4.3 out of 5 on a Markdown quiz after watching the AI-generated videos, suggesting short-term learning — a narrower finding that bears on this claim
• Design effects from aggregation are tolerable with moderate numbers of groups and few covariates — related
• Differential rapid guessing often resulted in test events with meaningfully distorted item content representation — related
• Distortive effects of disengagement on school means were diluted by the high proportion of students exhibiting no non-effortful behavior — a narrower finding that bears on this claim
• Computer-based testing's routine capture of item-level response times has driven increased interest in applications of response time to measuring ability and achievement — related