When well implemented, RD and experimental estimators produce impact estimates that are not significantly different and similar in magnitude on average¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2018 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? In a synthetic within-study comparison, RD impact estimates are not significantly different from experimental estimates of the same intervention and have a similar magnitude on average. → Philip Gleason 2018

Evidence¶

Philip Gleason 2018¶

Philip Gleason, Alexandra Resch, Jillian Berk. (2018). RD or Not RD: Using Experimental Studies to Assess the Performance of the Regression Discontinuity Approach. Evaluation Review, vol. 42. https://www.mathematica.org/publications/rd-or-not-rd-using-experimental-studies-to-assess-the-performance-of-the-regression-discontinuity doi:10.1177/0193841x18787267

q2 · i? · causal · r2

Synthetic within-study comparison generating RD datasets from two experimental evaluations of educational interventions (Educational Technology Study and Teach for America Study) and comparing RD impact estimates with the experimental estimates of the same interventions. The article reports estimates that are "not significantly different from one another" with similar average magnitude; no effect size is printed.

"RD and experimental estimators produce impact estimates that are not significantly different from one another and have a similar magnitude, on average."

Discussion¶

Related Claims¶

• Regression discontinuity impact estimates for two education interventions were not significantly different from experimental impact estimates, though point-estimate differences were sometimes nontrivial — a narrower finding that bears on this claim
• Regression discontinuity estimates show high internal validity, with average bias below 0.01 standard deviations relative to RCT estimates at the same cutoff — a narrower finding that bears on this claim
• Shrunken study-specific RD bias estimates stay below 0.07 standard deviations, indicating good external validity — a narrower finding that bears on this claim
• Individual RD impact estimates show substantial variation in bias, warranting caution when interpreting single RD estimates — related
• Robust RDD methods protect against model misspecification but require much larger sample sizes to detect small effects — related