Rothstein falsification test

Research Method · All research methods
Evidence · 8 claims (1 for, 2 mixed, 5 against) · 2 studies (2 theoretical), q1–q2 · 0 of 2 report an effect size · 8 claims rest on one study

Description

A falsification test for teacher value-added models, introduced by Jesse Rothstein (2010) and re-assessed in this paper. It rests on the observation that standard VAMs "suggest implausible future teacher effects on past student achievement, a finding that obviously cannot be viewed as causal", which "appears to indicate bias in VAM estimates of current teacher contributions to student learning". The present paper evaluates whether a failed test truly demonstrates bias.

Accounts

• Rothstein falsification test for teacher value-added models: A falsification test for teacher value-added models, introduced by Jesse Rothstein (2010) and re-assessed in this paper. It rests on the observation that standard VAMs "suggest implausible future teacher effects on past student achievement, a finding that obviously cannot be viewed as causal", which "appears to indicate bias in VAM estimates of current teacher contributions to student learning". The present paper evaluates whether a failed test truly demonstrates bias. (Dan Goldhaber (2012))
• The Rothstein falsification test as a diagnostic for value-added model bias: The Rothstein falsification test is a diagnostic procedure, proposed by Jesse Rothstein (2010), that regresses students' past achievement on future teacher assignments; implausibly large future teacher effects are taken to indicate bias in typical value-added model estimates of teacher contributions to student learning on standardized tests. This article treats the test as an inferential tool whose signal can arise without bias, since "tracking could be based on lagged student achievement, which is usually accounted for in VAMs." It is used here as the object of a methodological critique rather than as a validated measure of estimator bias. (Dan Goldhaber (2015))

Claims

• Rothstein (2010) found that standard VAMs imply implausible future-teacher effects on past student achievement [+W]
• The Rothstein falsification test is not definitive in showing bias in VAM teacher effect estimates [-W]
• The Rothstein test can reject VAMs even when students are randomly assigned conditional on the covariates in the model [-W]
• The Rothstein falsification test can reject value-added models even when estimated teacher effects are unbiased [-W]
• The Rothstein falsification test generates large and implausible future teacher effects even when students are randomly assigned to classrooms [-W] — (also cited [+M])
• Theoretically and in simulations, the Rothstein test often falsifies unbiased VAMs and fails to falsify biased VAMs [-W] — (also cited [+M])
• Tracking based on lagged student achievement is usually accounted for in value-added models, which is why the falsification test can mislead [~W] — (also cited [-W])
• Detecting that students are tracked to teachers does not necessarily imply bias in teacher effectiveness estimates [~W] — (also cited [-W])

Related Research Methods

-

Key Sources

• Dan Goldhaber, Duncan Chaplin. (2012). Assessing the Rothstein Test: Does It Really Show Teacher Value-Added Models Are Biased? Working Paper 5. Seattle, WA: Center for Education Data & Research, University of Washington. https://www.mathematica.org/publications/assessing-the-rothstein-test-does-it-really-show-teacher-valueadded-models-are-biased
• Dan Goldhaber, Duncan Dunbar Chaplin. (2015). Assessing the "Rothstein Falsification Test": Does It Really Show Teacher Value-Added Models Are Biased? Journal of Research on Educational Effectiveness, vol. 8, issue 1. https://www.mathematica.org/publications/assessingthe-rothstein-falsification-test-does-it-really-show-teacher-valueadded-models-are-biased

Page metadata

Field	Value
type	research-method
id	rothstein-falsification-test
title	Rothstein falsification test
description	A falsification test for teacher value-added models, introduced by Jesse Rothstein (2010) and re-assessed in this paper.
status	review
moved_from	element/rothstein-falsification-test-vam, theory/rothstein-falsification-test-vam-diagnostic
generated	by process:sweep-kinds on 2026-10-10
trust	unverified — no one has confirmed this page's content
sources	2 — dan-goldhaber-2012, dan-goldhaber-2015

Frontmatter is the page's machine-readable record — see the Schema & Guide.