Theoretically and in simulations, the Rothstein test often falsifies unbiased VAMs and fails to falsify biased VAMs¶

Claim · All claims
Evidence · 1 study · theoretical r? · q2
Currency · newest study 2015 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2018: tool and model accuracy changes quickly

Subclaims¶

q2 i? Theoretical analysis and simulations show the Rothstein test will often falsify VAMs that are unbiased and fail to falsify VAMs that are biased. → Dan Goldhaber 2015

Evidence¶

Dan Goldhaber 2015¶

Dan Goldhaber, Duncan Dunbar Chaplin. (2015). Assessing the "Rothstein Falsification Test": Does It Really Show Teacher Value-Added Models Are Biased? Journal of Research on Educational Effectiveness, vol. 8, issue 1. https://www.mathematica.org/publications/assessingthe-rothstein-falsification-test-does-it-really-show-teacher-valueadded-models-are-biased doi:10.1080/19345747.2014.978059

q2 · i? · theoretical · r?

Simulation and theoretical results reported by the authors demonstrating the test can both "falsify VAMs that are unbiased" and "fail to falsify VAMs that are biased." No effect sizes or simulation counts are printed on this page.

"Specifically, we show theoretically and through simulations that the test proposed in Rothstein will often falsify VAMs that are unbiased and fail to falsify VAMs that are biased."

Discussion¶

Related Claims¶

• Tracking based on lagged student achievement is usually accounted for in value-added models, which is why the falsification test can mislead — a narrower finding that bears on this claim
• Detecting that students are tracked to teachers does not necessarily imply bias in teacher effectiveness estimates — a narrower finding that bears on this claim
• The Rothstein falsification test generates large and implausible future teacher effects even when students are randomly assigned to classrooms — a narrower finding that bears on this claim
• Rothstein (2010) found that standard VAMs imply implausible future-teacher effects on past student achievement — related
• The Rothstein falsification test is not definitive in showing bias in VAM teacher effect estimates — a broader claim this one bears on
• The Rothstein test can reject VAMs even when students are randomly assigned conditional on the covariates in the model — related
• The Rothstein falsification test can reject value-added models even when estimated teacher effects are unbiased — possibly the same claim (merge candidate)