Removing disengaged examinees from the sample will likely induce bias in estimates of educational effectiveness¶

Claim · All claims
Evidence · 1 study · associational r? · q3
Currency · newest study 2019 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q3 i? Removing disengaged test takers from the sample will likely induce bias in estimates of school contributions to growth, achievement gaps, and summer learning loss. → Megan Kuhfeld 2019

Evidence¶

Megan Kuhfeld 2019¶

Megan Kuhfeld, James Soland. (2019). Using assessment metadata to quantify the impact of test disengagement on estimates of educational effectiveness. Journal of Research on Educational Effectiveness 13(1), 147-175. https://www.nwea.org/research/publication/using-assessment-metadata-to-quantify-the-impact-of-test-disengagement-on-estimates-of-educational-effectiveness/ doi:10.1080/19345747.2019.1636437

q3 · i? · associational · r?

The study's reported result on the sample-removal technique: the authors state that "removing disengaged examinees from the sample will likely induce bias in the estimates". No effect size is printed in the available text.

"Our results indicate that removing disengaged examinees from the sample will likely induce bias in the estimates, although as a whole accounting for disengagement had minimal impact on the metrics we examined."

Discussion¶

Related Claims¶

• Assessment metadata can be used to quantify the impact of test disengagement on estimates of educational effectiveness — related
• Accounting for test disengagement had minimal impact on estimates of school growth contributions, achievement gaps, and summer learning loss — related
• Summer learning loss estimates changed significantly under filtering but not effort-moderated scoring — related
• Filtered students had much lower effort-moderated achievement, suggesting motivation filtering may bias mean estimates — related
• Student test disengagement may bias estimates of educational effectiveness based on observed test results — related
• Restricting a sample to students who used the treatment can block a mediating path and induce overcontrol bias, attenuating estimated effects — a narrower finding that bears on this claim
• After accounting for publication bias, the estimated true effect favoring PS-I is strong (Hedge's g = 0.87) — related
• Rapid guessing had mostly minimal effects on estimates of school contributions to student growth in this sample — reports the opposite
• Motivation filtering reversed the estimated Black-White fall-to-fall growth gap direction in later grades — a narrower finding that bears on this claim