Teacher value-added estimates are highly correlated across model specifications that differ in student and peer control variables¶

Claim · All claims
Evidence · 1 study · associational r? · q2
Currency · newest study 2015 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2018: tool and model accuracy changes quickly

Subclaims¶

q2 i? Teacher effectiveness estimates are highly correlated across VAM specifications varying in background controls and double-lagged scores. → Matthew T. Johnson 2015

Evidence¶

Matthew T. Johnson 2015¶

Matthew T. Johnson, Stephen Lipscomb, Brian Gill. (2015). Sensitivity of Teacher Value-Added Estimates to Student and Peer Control Variables. Journal of Research on Educational Effectiveness, vol. 8, issue 1. https://www.mathematica.org/publications/journal-article-sensitivity-of-teacher-valueadded-estimates-to-student-and-peer-control-variables

q2 · i? · associational · r?

Observational analysis of teacher value-added estimates under multiple model specifications using data from a northern state. The article reports that "teacher effectiveness estimates are highly correlated across specifications." No correlation coefficient or effect size is printed in the available text.

"Using data from a northern state, we find that teacher effectiveness estimates are highly correlated across specifications."

Discussion¶

Related Claims¶

• Up to 26% of teachers in the bottom value-added quintile under one specification are ranked higher under another specification — a narrower finding that bears on this claim
• The working paper examines the sensitivity and precision of teacher value-added estimates under specifications differing in student-level, peer-level, and double-lagged achievement controls — possibly the same claim (merge candidate)
• The paper tests two previously unevaluated VAM model variations: teacher-year level average peer characteristics and demographic variation in the lagged-achievement relationship — related
• Replacing classroom peer characteristics with teacher-level averages and allowing demographics to influence the lagged-achievement relationship have smaller practical consequences than other specification choices — related
• Teachers in districts with large fractions of disadvantaged students receive lower value-added ratings when student background characteristics are omitted — related
• The choice of which assessment to use as the outcome measure matters more for value-added estimates than any student or peer control-variable specification choice — related
• Mathematica developed and estimated VAMs for use in Phase 1 of the Pennsylvania evaluation pilot — related
• Pretest-posttest correlation coefficients for state assessment data vary by student achievement level (low-performing, average-performing, proficient). — related
• Alternative value-added specifications produce substantially different teacher estimates but small changes in distribution tails — reports the opposite
• Disadvantaged students in grades 4-8 receive less-effective teaching on average than other students in the same grades — related
• Value-added teacher rankings are insensitive to correcting measurement error in the pretest, with correlations of 0.91–0.99 for elementary and 0.77–0.98 for middle school teachers — related
• The Memphis value-added model estimates school effects on student performance while adjusting for mobility, demographics, and measurement error — related
• Simulated adjustment of ratings for classroom context improves Black teachers' rankings by about 8 percentile points, with the largest gains for teachers serving more disadvantaged students — related
• Teachers with the same observation scores can have very different value-added scores, with considerable overlap across observation-score quintiles — related
• Models with similar rank correlations differ substantially in estimating difficulty magnitude (RMSE and MAE) — related
• The Pittsburgh VAMs incorporate course-specific assessments, attendance, and course completion alongside state assessments — related
• Omitting same-subject pre-tests affects value-added estimates more than excluding other student background characteristics — related