Disengaged test taking materially impacts individual state summative test scores but may have relatively minor impact on score aggregations¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Disengagement materially affects individual state summative test scores while its impact on score aggregations may be relatively minor. → Steven Wise 2021

Evidence¶

Steven Wise 2021¶

Steven Wise, Jonghwan (Jay) Lee, Sukkeun Im. (2021). The impact of disengaged test taking on a state's accountability test results. Educational Assessment, 26(3), 163-174. https://www.nwea.org/research/publication/the-impact-of-disengaged-test-taking-on-a-states-accountability-test-results/ doi:10.1080/10627197.2021.1956897

q2 · i? · associational · r2

Overall conclusion of the study of the Spring 2019 state summative assessment: disengagement has "a material impact on individual state summative test scores" while aggregation-level impact "may be relatively minor". No effect sizes are printed in the abstract.

"Overall, results of this study indicate that disengagement has a material impact on individual state summative test scores, though its impact on score aggregations may be relatively minor."

Discussion¶

Related Claims¶

• Disengaged test taking in international assessment programs threatens the validity of inferences about a country's educational attainment — related
• Test-taking disengagement distorts both individual and aggregated test scores — reports the opposite
• Disengagement had only minimal impact on school mean performance, proficiency rates, and relative ranking on the state accountability test — a narrower finding that bears on this claim
• Two decades of research have produced advances in both the measurement of test-taking disengagement and understanding of its distortive impact on individual and aggregated scores — a broader claim this one bears on
• Disengagement rates on the state summative test varied materially across gender, ethnicity, IEP status, LEP status, FRL status, and disability status — related
• A meaningful percentage of Grade 8 state summative test events showed rapid guessing, varying by subject: 5.5% in mathematics, 6.7% in ELA, and 3.5% in science — related
• Disengaged test-taking responses can distort the content representation of test items — related
• Twenty years of research on disengaged test taking in international assessment programs yields six insights with implications for assessment programs — related
• Test-taking engagement plays an essential role in the pursuit of valid scores — a broader claim this one bears on
• Test disengagement could undermine inferences drawn from observed achievement scores — related
• Very short response times (rapid guessing) indicate disengaged test taking in both high-stakes and low-stakes testing contexts — related
• Rapid-guessing behavior varies by item and is significantly related to item length, item position, and presence of ancillary reading material — related
• Rapid guessing tends to negatively distort scores and diminish validity — related
• Rapid-guessing behavior on computer-based tests signals disengaged test taking that threatens score validity — related
• Rapid guessing is a validated indicator of disengaged item responding — related
• Technology-enhanced assessment items were studied as a potential means of increasing student test engagement and reducing disengagement — related
• Summative examinations have a dual function: measuring achieved learning while also shaping and supporting learning through engagement with tasks — related
• Even a small amount of rapid-guessing behavior can impact institutional rankings — related
• Previous research found a meaningful correlation between countries' mean engagement and mean performance on large-scale international assessments, motivating validity questions about aggregated scores — related
• Rapid-guessing behavior occurs on only about 1% of item responses in a commercially available low-stakes assessment — related
• Computer-based testing's routine capture of item-level response times has driven increased interest in applications of response time to measuring ability and achievement — related
• Disengagement produced both positive and negative distortion of individual student scores, which tended to cancel out much of the net distortive effect on school means — related
• Opting out has disadvantages: it can obscure implicit knowledge, enable excessive opting out, and complicate assessment — related