The Rothstein falsification test can reject value-added models even when estimated teacher effects are unbiased¶

Claim · All claims
Evidence · 1 study · theoretical r? · q1
Currency · newest study 2012 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2015: tool and model accuracy changes quickly

Subclaims¶

q1 i? Under plausible conditions, the Rothstein falsification test rejects VAMs even when there is no bias in estimated teacher effects. → Dan Goldhaber 2012

Evidence¶

Dan Goldhaber 2012¶

Dan Goldhaber, Duncan Chaplin. (2012). Assessing the Rothstein Test: Does It Really Show Teacher Value-Added Models Are Biased? Working Paper 5. Seattle, WA: Center for Education Data & Research, University of Washington. https://www.mathematica.org/publications/assessing-the-rothstein-test-does-it-really-show-teacher-valueadded-models-are-biased

q1 · i? · theoretical · r?

Theoretical illustration plus simulation study by the authors showing conditions under which the Rothstein test rejects VAMs despite "no bias in estimated teacher effects". No effect sizes are printed in the available text.

"we illustrate—theoretically and through simulations—plausible conditions under which the Rothstein falsification test rejects VAMs even when there is no bias in estimated teacher effects"

Discussion¶

Related Claims¶

• Rothstein (2010) found that standard VAMs imply implausible future-teacher effects on past student achievement — related
• The Rothstein falsification test is not definitive in showing bias in VAM teacher effect estimates — possibly the same claim (merge candidate)
• The Rothstein test can reject VAMs even when students are randomly assigned conditional on the covariates in the model — related
• The Rothstein falsification test generates large and implausible future teacher effects even when students are randomly assigned to classrooms — a narrower finding that bears on this claim
• Theoretically and in simulations, the Rothstein test often falsifies unbiased VAMs and fails to falsify biased VAMs — possibly the same claim (merge candidate)
• Detecting that students are tracked to teachers does not necessarily imply bias in teacher effectiveness estimates — related
• CDM-AIR declares four falsification conditions, including moderation nullity and context redundancy — a narrower finding that bears on this claim
• Tracking based on lagged student achievement is usually accounted for in value-added models, which is why the falsification test can mislead — a narrower finding that bears on this claim