NWEA research brief on measuring the impact of test disengagement on effectiveness estimates¶

Element · All elements
Evidence · 4 claims (4 for) · 2 studies (2 associational), q1–q3 · 0 of 2 report an effect size · 4 claims rest on one study

Description¶

A 2019 NWEA research brief by Megan Kuhfeld and James Soland, categorized under Measurement & scaling, School & test engagement, and Student growth & accountability policies. Its stated purpose is to help readers "Learn more about our examination of student disengagement and how it may bias estimates of effectiveness based on observed test results." The brief page also links associated research, including a working paper and journal article on using assessment metadata to quantify disengagement's impact.

Design Implications¶

Context¶

Requirements¶

• Readers seeking the underlying analyses must consult the associated working paper or journal article the brief lists

Constraints¶

• The page as supplied prints only a summary description and links, not the brief's full analyses

Target Learners¶

• K-12 students taking MAP Growth-style assessments

Target Learning Goals¶

• Accurate interpretation of test-based estimates of educational effectiveness

Claims¶

• Assessment metadata can be used to quantify the impact of test disengagement on estimates of educational effectiveness [+W]
• Student test disengagement may bias estimates of educational effectiveness based on observed test results [+W]
• Accounting for test disengagement had minimal impact on estimates of school growth contributions, achievement gaps, and summer learning loss [+W]
• Removing disengaged examinees from the sample will likely induce bias in estimates of educational effectiveness [+W]

Related Elements¶

• Two techniques for accounting for test disengagement: sample removal and rapid-guess score adjustment

Examples¶

• Provide guidance to policy makers and evaluators on accounting for disengagement in their own work

Key Sources¶

• Megan Kuhfeld, James Soland. (2019). Measuring the impact of test disengagement on estimates of educational effectiveness. NWEA Research. https://www.nwea.org/research/publication/measuring-the-impact-of-test-disengagement-on-estimates-of-educational-effectiveness/