The working paper examines the sensitivity and precision of teacher value-added estimates under specifications differing in student-level, peer-level, and double-lagged achievement controls¶

Claim · All claims
Evidence · 1 study · associational r? · q2
Currency · newest study 2013 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2016: tool and model accuracy changes quickly

Subclaims¶

q2 i? Teacher value-added model estimates are examined for sensitivity and precision across model specifications that vary in whether they include student-level background characteristics, peer-level background characteristics, and/or a double-lagged achievement score. → Matthew Johnson 2013

Evidence¶

Matthew Johnson 2013¶

Matthew Johnson, Stephen Lipscomb, Brian Gill. (2013). Sensitivity of Teacher Value-Added Estimates to Student and Peer Control Variables. Working Paper 25. Cambridge, MA: Mathematica Policy Research. https://www.mathematica.org/publications/sensitivity-of-teacher-valueadded-estimates-to-student-and-peer-control-variables

q2 · i? · associational · r?

The publication abstract states the paper's purpose: examining sensitivity and precision of teacher value-added model estimates under specifications that differ in included controls. The landing page prints no results, sample, or statistics; only the stated aim is available.

"This working paper examines the sensitivity and precision of teacher value-added model (VAM) estimates obtained under model specifications that differ based on whether they include student-level background characteristics, peer-level background characteristics, and/or a double-lagged achievement score."

Discussion¶

Related Claims¶

• Up to 26% of teachers in the bottom value-added quintile under one specification are ranked higher under another specification — a narrower finding that bears on this claim
• Teacher value-added estimates are highly correlated across model specifications that differ in student and peer control variables — possibly the same claim (merge candidate)
• The choice of which assessment to use as the outcome measure matters more for value-added estimates than any student or peer control-variable specification choice — a narrower finding that bears on this claim
• Replacing classroom peer characteristics with teacher-level averages and allowing demographics to influence the lagged-achievement relationship have smaller practical consequences than other specification choices — related
• The paper tests two previously unevaluated VAM model variations: teacher-year level average peer characteristics and demographic variation in the lagged-achievement relationship — related
• Teachers of students with hard-to-predict achievement levels tend to have less precise value-added estimates than teachers of other students — related
• Empirical Bayes shrinkage increases the precision of value-added estimates and reduces their absolute value for teachers of hard-to-predict students — related
• Teachers in districts with large fractions of disadvantaged students receive lower value-added ratings when student background characteristics are omitted — related
• Including track indicators and classroom achievement controls simultaneously reduces precision of value-added estimates — a narrower finding that bears on this claim
• Mathematica developed and estimated VAMs for use in Phase 1 of the Pennsylvania evaluation pilot — related
• Alternative value-added specifications produce substantially different teacher estimates but small changes in distribution tails — related
• Bias, not precision loss, drives the difference in proxy pre-test value-added estimates — related
• Omitting same-subject pre-tests affects value-added estimates more than excluding other student background characteristics — a narrower finding that bears on this claim
• Most differences in evaluation scores between the Colorado Growth Model and value added are not related to the characteristics of students' teachers — related
• Oklahoma's 2013-2014 teacher value-added estimates combined test scores with other information about teachers and students — related
• The Rothstein falsification test is not definitive in showing bias in VAM teacher effect estimates — related
• Successive cohort indicators are subject to false performance gains, and the paper assesses which school evaluation methods provide useful accountability information — related
• Value-added teacher rankings are insensitive to correcting measurement error in the pretest, with correlations of 0.91–0.99 for elementary and 0.77–0.98 for middle school teachers — related
• The Memphis value-added model estimates school effects on student performance while adjusting for mobility, demographics, and measurement error — related
• The number of schools in a study would have to nearly double to compensate for the precision lost by using school-level proficiency instead of student-level data — related
• Overall false positive and false negative error rates in teacher value-added performance classification are about 10 and 20 percent, respectively — related
• Evidence is mixed on whether including demographic variables as predictors improves model performance in educational prediction — related
• Academic and behavioral classroom factors, not student demographics, drive the effect of composition on evaluator ratings — related
• Simulated adjustment of ratings for classroom context improves Black teachers' rankings by about 8 percentile points, with the largest gains for teachers serving more disadvantaged students — related
• Teachers with the same observation scores can have very different value-added scores, with considerable overlap across observation-score quintiles — related
• Rapid guessing did not appear to substantively affect teacher value-added estimates under the value-added specifications used — a narrower finding that bears on this claim
• Race-specific value-added scores predict teachers' subsequent effects: Black students taught by a teacher with high prior effectiveness for Black students show gains of about one standard deviation when the teacher moves schools or grades — related
• Overall false positive error rates are about 10 percent and false negative error rates about 20 percent for value-added performance classifications — related
• Proxy pre-test value-added estimates show bias, but may still carry information about teacher effectiveness — related
• Measurement error in the mediator reduces statistical power for mediator-achievement association estimates — related
• Including track indicators in value-added models may reduce tracking-related bias for high school reading teachers — related
• Accounting for mean and standard deviation of classroom achievement may reduce bias for middle school math teachers — a narrower finding that bears on this claim
• Error rates for comparing a teacher's performance to the average are about 25 percent with three years of data and 35 percent with one year of data — related
• Variance terms in multi-armed RCT estimators need slight adjustment under the finite-population framework, which can reduce precision — related
• Some educators are concerned that students scoring at the bottom or top of a test's score distribution will negatively affect the value-added estimates of those students' teachers — related