Robust RDD methods protect against model misspecification but require much larger sample sizes to detect small effects¶

Claim · All claims
Evidence · 1 study · theoretical r2 · q2
Currency · newest study 2017 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q2 i? State-of-the-art statistical methods for regression discontinuity designs can protect against inaccuracies from incorrect regression models, but detecting small effects with these methods requires much larger sample sizes. → John Deke 2017

Evidence¶

John Deke 2017¶

John Deke, Thomas Wei, Tim Kautz. (2017). Asymdystopia: The Threat of Small Biases in Evaluations of Education Interventions that Need to be Powered to Detect Small Impacts. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://www.mathematica.org/publications/asymdystopia-the-threat-of-small-biases-in-evaluations-of-education-interventions-that-need

q2 · i? · theoretical · r2

The paper's key finding on regression discontinuity designs: robust methods guard against "inaccuracies from incorrect regression models," but "much larger sample sizes are needed in order to detect small effects when using these methods." No effect size is printed.

"For RDDs, state-of-the-art statistical methods can protect against inaccuracies from incorrect regression models, but this protection comes at a cost – much larger sample sizes are needed in order to detect small effects when using these methods."

Discussion¶

Related Claims¶

• Education evaluators increasingly design studies to detect impacts smaller than 0.20 standard deviations — related
• A regression discontinuity design on Review Exercise data supports a causal link between leveling up and higher assessment accuracy, at least for the first level-up — related
• A regression discontinuity study found a positive but non-significant effect of +0.16 on standardized literacy growth — related
• Regression discontinuity impact estimates for two education interventions were not significantly different from experimental impact estimates, though point-estimate differences were sometimes nontrivial — related
• The RD evaluation was powered to detect overall SES impacts of 0.12 standard deviations or larger — a narrower finding that bears on this claim
• Individual RD impact estimates show substantial variation in bias, warranting caution when interpreting single RD estimates — related
• The large sample requirement of RD designs is driven primarily by bandwidth selection, not by adjusting for random misspecification error — reports the opposite
• Clustered regression discontinuity designs typically require three to four times larger samples than clustered experimental designs to produce impact estimates with the same level of statistical precision in education evaluations — related
• Small sample sizes reduced the power of the school choice analysis to detect achievement effects — related
• In Clark County, One-Star designation significantly raises math and ELA scores for all students — a narrower finding that bears on this claim
• When well implemented, RD and experimental estimators produce impact estimates that are not significantly different and similar in magnitude on average — related
• Author-reported RD findings count toward effectiveness ratings only if standard errors account for clustering of students at unique forcing variable values — related
• Most studies of literacy resilience have (very) small sample sizes and low statistical power to find relevant effects — related