Individual RD impact estimates show substantial variation in bias, warranting caution when interpreting single RD estimates¶

Claim · All claims
Evidence · 1 study · quant-synthesis r? · q3
Currency · newest study 2018 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? With unshrunken estimates, the mean RD bias is essentially zero but the distribution of bias estimates is less tight, especially with smaller samples and when parametric RD analyses are used. → Duncan D. Chaplin 2018

Evidence¶

Duncan D. Chaplin 2018¶

Duncan D. Chaplin, Thomas D. Cook, Jelena Zurovac, Jared S. Coopersmith, Mariel M. Finucane, Lauren N. Vollmer, Rebecca E. Morris. (2018). The Internal and External Validity of the Regression Discontinuity Design: A Meta-Analysis of 15 Within-Study Comparisons. Journal of Policy Analysis and Management, vol. 37, issue 2. https://www.mathematica.org/publications/the-internal-and-external-validity-of-the-regression-discontinuity-design-a-meta-analysis-of-15 doi:10.1002/pam.22051

q3 · i? · quant-synthesis · r?

Distributional analysis of unshrunken individual impact estimates across the 15 within-study comparisons. The mean bias is "still essentially zero," but the distribution is less tight with smaller samples and parametric analyses.

"With unshrunken estimates, the mean RD bias is still essentially zero, but the distribution of RD bias estimates is less tight, especially with smaller samples and when parametric RD analyses are used."

Discussion¶

Related Claims¶

• Manipulation of the assignment variable can substantially influence RD impact estimates, particularly when manipulation is related to the outcome and occurs close to the cutoff — a narrower finding that bears on this claim
• Regression discontinuity estimates show high internal validity, with average bias below 0.01 standard deviations relative to RCT estimates at the same cutoff — related
• Robust RDD methods protect against model misspecification but require much larger sample sizes to detect small effects — related
• Regression discontinuity impact estimates for two education interventions were not significantly different from experimental impact estimates, though point-estimate differences were sometimes nontrivial — related
• The meta-analysis finds some evidence favoring non-parametric RD methods over parametric ones — related
• Shrunken study-specific RD bias estimates stay below 0.07 standard deviations, indicating good external validity — related
• Schools that missed AYP showed some positive subsequent achievement impacts under NCLB accountability — related
• Regression discontinuity estimates of EL status effects on SPED placement consistently differ substantively from regression analysis results — related
• When well implemented, RD and experimental estimators produce impact estimates that are not significantly different and similar in magnitude on average — related
• Accountability sanction effects were not consistent across years and outcomes — related