Author-reported RD findings count toward effectiveness ratings only if standard errors account for clustering of students at unique forcing variable values¶

Claim · All claims
Evidence · 1 study · theoretical r3 · q2
Currency · newest study 2010 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Author-reported findings are included in effectiveness ratings only when study authors account for clustering of students at unique values of the forcing variable; otherwise the WWC will not rely on the statistical significance of the findings. → Schochet 2010

Evidence¶

Schochet 2010¶

Schochet, P., Cook, T., Deke, J., Imbens, G., Lockwood, J.R., Porter, J., Smith, J. (2010). Standards for Regression Discontinuity Designs. Retrieved from What Works Clearinghouse website: http://ies.ed.gov/ncee/wwc/pdf/wwc_rd.pdf.

q2 · i? · theoretical · r3

The panel's reporting requirement for discrete forcing variables states that if authors "do not account for clustering," the WWC "will not rely on the statistical significance of the findings" for effectiveness ratings, though findings may still be included as substantively important.

"If the authors do not account for clustering, then t he WWC will not rely on the statistical significance of the findings fro m the study . However, the findings can still be included as “substantively important”"

Discussion¶

Related Claims¶

• A study qualifies as an RD design only if assignment uses an ordinal forcing variable with at least four unique values on each side of the cutoff and no confounding factor — related
• Standard errors sometimes differ across estimators, so policy conclusions from clustered education RCTs could be sensitive to the choice of estimator — related
• Author-reported statistically significant program effects did not remain significant after WWC corrections for clustering and multiple comparisons — related
• Regression discontinuity impact estimates for two education interventions were not significantly different from experimental impact estimates, though point-estimate differences were sometimes nontrivial — related
• Manipulation of the assignment variable can substantially influence RD impact estimates, particularly when manipulation is related to the outcome and occurs close to the cutoff — related
• Statistical theory predicts that regression discontinuity provides valid causal inference at the cutoff score determining treatment assignment — related
• Robust RDD methods protect against model misspecification but require much larger sample sizes to detect small effects — related