The Rothstein falsification test is not definitive in showing bias in VAM teacher effect estimates¶

Claim · All claims
Evidence · 1 study · theoretical r? · q1
Currency · newest study 2012 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2015: tool and model accuracy changes quickly

Subclaims¶

q1 i? On the whole, the authors' findings show the Rothstein falsification test is not definitive in showing bias in VAM estimates. → Dan Goldhaber 2012

Evidence¶

Dan Goldhaber 2012¶

Dan Goldhaber, Duncan Chaplin. (2012). Assessing the Rothstein Test: Does It Really Show Teacher Value-Added Models Are Biased? Working Paper 5. Seattle, WA: Center for Education Data & Research, University of Washington. https://www.mathematica.org/publications/assessing-the-rothstein-test-does-it-really-show-teacher-valueadded-models-are-biased

q1 · i? · theoretical · r?

Authors' overall conclusion from their theoretical and simulation work: the test "is not definitive in showing bias". This is an interpretation of their own analyses, with no effect sizes printed in the available text.

"On the whole, our findings show that the "Rothstein falsification test" is not definitive in showing bias, which suggests a much more encouraging picture for those wishing to use VAM teacher effect estimates for policy purposes"

Discussion¶

Related Claims¶

• Rothstein (2010) found that standard VAMs imply implausible future-teacher effects on past student achievement — related
• The Rothstein falsification test can reject value-added models even when estimated teacher effects are unbiased — possibly the same claim (merge candidate)
• Detecting that students are tracked to teachers does not necessarily imply bias in teacher effectiveness estimates — possibly the same claim (merge candidate)
• Theoretically and in simulations, the Rothstein test often falsifies unbiased VAMs and fails to falsify biased VAMs — a narrower finding that bears on this claim
• The Rothstein falsification test generates large and implausible future teacher effects even when students are randomly assigned to classrooms — a narrower finding that bears on this claim
• Tracking based on lagged student achievement is usually accounted for in value-added models, which is why the falsification test can mislead — a narrower finding that bears on this claim
• The Rothstein test can reject VAMs even when students are randomly assigned conditional on the covariates in the model — a narrower finding that bears on this claim
• The working paper examines the sensitivity and precision of teacher value-added estimates under specifications differing in student-level, peer-level, and double-lagged achievement controls — related