Evaluators should re-estimate IRT scores treating rapid guesses as missing and compare with unadjusted results¶

Strategy · All strategies
Evidence · 1 claim (1 for) · 1 study (1 associational), q2 · 0 of 1 report an effect size · 1 claim rests on one study

Description¶

For evaluators with appropriate computer-based test metadata, the paper recommends a sensitivity workflow: set response-time thresholds, then apply "the most defensible and conservative approach is to re-estimate IRT-based scores treating rapid guessing as missing, use those scores in models, and compare results." Motivation filtering can serve as a quicker check, moving to effort-moderated scoring if results are sensitive. The paper also suggests a rough check of the proportion of items with response times under ten seconds, clustered by student, treatment condition, and subgroup.

Design Implications¶

Context¶

Requirements¶

• Access to a CBT with item-level response time metadata
• Sufficient test length to remove rapid-guessed items without excessive imprecision

Constraints¶

• Filtering can upwardly bias mean achievement estimates if rapid guessing correlates with true achievement
• Many evaluators lack a CBT, item-level data, or associated metadata

Target Learners¶

• K-8 students in program evaluation and accountability contexts

Target Learning Goals¶

• Unbiased estimates of school contributions to growth, achievement gaps, and summer learning loss

Affordances¶

• Rapid Guessing Response Time Threshold Framework

Claims¶

• Motivation filtering reversed the estimated Black-White fall-to-fall growth gap direction in later grades [+W]

Related Strategies¶

• Use survey and assessment disengagement metadata as a new source of SEL data for practice, policy, and research

Examples¶

-

Key Sources¶

• Kuhfeld, M. & Soland, J. (2019). Using Assessment Metadata to Quantify the Impact of Test Disengagement on Estimates of Educational Effectiveness. (The Collaborative for Student Growth at NWEA Working Paper). https://www.nwea.org/research/publication/using-assessment-metadata-to-quantify-the-impact-of-test-disengagement-on-estimates-of-educational-effectiveness-paper/