Rapid guessing had mostly minimal effects on estimates of school contributions to student growth in this sample¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q2
Currency · newest study 2019 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2022: tool and model accuracy changes quickly

Subclaims¶

q2 i? Only the fall-to-fall linear growth difference between unadjusted and filtered models was significant at the .05 level; other parameter differences were not significant and magnitudes were modest. → Kuhfeld 2019 q2 i? Empirical Bayes school-level estimates from original and adjusted scores were highly correlated (filtering .914 to .978; effort-moderated roughly at or above .99). → Kuhfeld 2019 (2)

Evidence¶

Kuhfeld 2019¶

Kuhfeld, M. & Soland, J. (2019). Using Assessment Metadata to Quantify the Impact of Test Disengagement on Estimates of Educational Effectiveness. (The Collaborative for Student Growth at NWEA Working Paper). https://www.nwea.org/research/publication/using-assessment-metadata-to-quantify-the-impact-of-test-disengagement-on-estimates-of-educational-effectiveness-paper/

q2 · i? · associational · r2

Three-level HLMs of MAP Growth reading RIT scores for 22,055 students in 343 schools compared unadjusted, filtered, and effort-moderated outcomes. The article reports "significant at the .05 level" only for the filtered fall-to-fall linear growth contrast; linear growth estimates ranged from 6.25 to 6.67 points.

"differences in estimates of school contributions to fall-to-fall linear growth between the unadjusted and filtered models are significant at the .05 level. None of the other differences are significant."

Kuhfeld 2019 (2)¶

Kuhfeld, M. & Soland, J. (2019). Using Assessment Metadata to Quantify the Impact of Test Disengagement on Estimates of Educational Effectiveness. (The Collaborative for Student Growth at NWEA Working Paper). https://www.nwea.org/research/publication/using-assessment-metadata-to-quantify-the-impact-of-test-disengagement-on-estimates-of-educational-effectiveness-paper/

q2 · i? · associational · r2

Figure 1 scatterplots compare empirical Bayes school random-effect estimates from original RIT scores with filtered and effort-moderated estimates. The text states "correlations range from .914 to .978" under filtering and roughly .99 or above for effort-moderated scores.

"When filtering is used, correlations range from .914 to .978. For effort-moderated scores, correlations are all roughly at or above .99."

Discussion¶

Related Claims¶

• Accounting for test disengagement had minimal impact on estimates of school growth contributions, achievement gaps, and summer learning loss — possibly the same claim (merge candidate)
• Filtered students had much lower effort-moderated achievement, suggesting motivation filtering may bias mean estimates — reports the opposite
• Motivation filtering reversed the estimated Black-White fall-to-fall growth gap direction in later grades — related
• Differences in rapid guessing between subgroups generally do not impact substantive interpretations of achievement gaps — related
• Summer learning loss estimates changed significantly under filtering but not effort-moderated scoring — related
• More of the variance in linear growth lies at the school level for fall-to-spring than for spring-to-spring estimates, in some cases twice or three times as much — related
• Disengagement had only minimal impact on school mean performance, proficiency rates, and relative ranking on the state accountability test — related
• Removing disengaged examinees from the sample will likely induce bias in estimates of educational effectiveness — reports the opposite
• Assessment metadata can be used to quantify the impact of test disengagement on estimates of educational effectiveness — a broader claim this one bears on
• Rapid guessing did not appear to substantively affect teacher value-added estimates under the value-added specifications used — related