Empirical Bayes shrinkage could be used to address bias in proxy pre-test value-added estimates¶

Claim · All claims
Evidence · 1 study · associational r? · q2
Currency · newest study 2018 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2021: tool and model accuracy changes quickly

Subclaims¶

q2 i? Empirical Bayes shrinkage, typically used for precision, might also be used to address bias so proxy pre-test estimates receive more appropriate weights in teacher evaluations. → Elias Walsh 2018

Evidence¶

Elias Walsh 2018¶

Elias Walsh, Dallas Dotter, Albert Y. Liu. (2018). Can More Teachers Be Covered? The Accuracy, Credibility, and Precision of Value-Added Estimates with Proxy Pre-Tests. Working Paper 64. Princeton, NJ: Mathematica Policy Research. https://www.mathematica.org/publications/can-more-teachers-be-covered-the-accuracy-credibility-and-precision-of-value-added-estimates

q2 · i? · associational · r?

The authors propose, based on the Oklahoma analysis, that "empirical Bayes shrinkage... might also be used to address bias" in proxy pre-test value-added estimates. This is a forward-looking proposal, not a tested manipulation in the available text.

"empirical Bayes shrinkage, an approach typically used to address precision, might also be used to address bias so that value-added estimates that rely on proxy pre-tests can be given more appropriate weights in teachers' evaluations."

Discussion¶

Related Claims¶

• Empirical Bayes shrinkage increases the precision of value-added estimates and reduces their absolute value for teachers of hard-to-predict students — related
• Teachers of students with hard-to-predict achievement levels tend to have less precise value-added estimates than teachers of other students — related
• Shrinkage could differentially affect consequences for teachers of fewer students or students with hard-to-predict achievement — related
• Teachers of students with low prior achievement and who receive free lunch tend to have less precise value-added estimates — related
• Shrinkage has no statistically significant effect on the relative probability that teachers of hard-to-predict students receive extreme value-added ratings — related
• Shrinkage does not change accountability consequences for teachers of hard-to-predict students, since extreme-rating probabilities are unaffected — related
• Shrinkage had no statistically significant effect on the relative probability that teachers of hard-to-predict students received consequences — related
• Bias, not precision loss, drives the difference in proxy pre-test value-added estimates — related
• Proxy pre-test value-added estimates show bias, but may still carry information about teacher effectiveness — related
• The Teacher Team Method and Full Roster Method produce similar estimates of teacher value added — related
• The paper clarifies confusion about the use of value-added information in teacher evaluation — a broader claim this one bears on
• Multiple sources of student information can be accumulated to estimate teacher effectiveness — related
• Precision-weighted shrinkage in MAIHDA yields reliable conservative estimates for small strata and helps solve the multiple-testing problem — related