Summer learning loss estimates changed significantly under filtering but not effort-moderated scoring¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2019 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Average summer drop between 4th and 5th grade was 1.68 RIT points unadjusted, 1.31 filtered, and 1.47 effort-moderated; the change was significant only when filtering. → Kuhfeld 2019

Evidence¶

Kuhfeld 2019¶

Kuhfeld, M. & Soland, J. (2019). Using Assessment Metadata to Quantify the Impact of Test Disengagement on Estimates of Educational Effectiveness. (The Collaborative for Student Growth at NWEA Working Paper). https://www.nwea.org/research/publication/using-assessment-metadata-to-quantify-the-impact-of-test-disengagement-on-estimates-of-educational-effectiveness-paper/

q2 · i? · associational · r2

CP growth models (Table 5) estimated summer change between 4th and 5th grade under three adjustment approaches. The article reports "an average drop of 1.68 RIT points over the summer" unadjusted versus 1.31 filtered and 1.47 effort-moderated, significant only for filtering.

"students in the unadjusted sample have an average drop of 1.68 RIT points over the summer compared to an average drop of 1.31 RIT points in the filtered sample and 1.47 points in the effort-moderated model."

Discussion¶

Related Claims¶

• Accounting for test disengagement had minimal impact on estimates of school growth contributions, achievement gaps, and summer learning loss — related
• Filtered students had much lower effort-moderated achievement, suggesting motivation filtering may bias mean estimates — related
• Ignoring summer loss changes which schools are identified as low performers — related
• Rapid guessing had mostly minimal effects on estimates of school contributions to student growth in this sample — related
• Removing disengaged examinees from the sample will likely induce bias in estimates of educational effectiveness — related
• Motivation filtering reversed the estimated Black-White fall-to-fall growth gap direction in later grades — related
• Assessment metadata can be used to quantify the impact of test disengagement on estimates of educational effectiveness — a broader claim this one bears on