Clustering effects in education random-assignment trials vary by design but are typically large, requiring large school samples¶

Claim · All claims
Evidence · 2 studies · 2 theoretical r? · q2
Currency · newest study 2008 · 1 of 2 entries checked against their sources (automated) · not reviewed by a person

Subclaims¶

q2 i? Clustering effects vary by random-assignment design but are typically large, so large school samples are required for education trials. → Peter Z. Schochet 2008

q2 i? Clustering effects vary by design but are typically large in evaluations of education programs using random assignment. → Peter Z. Schochet 2005

Evidence¶

Peter Z. Schochet 2008¶

Peter Z. Schochet. (2008). Statistical Power for Random Assignment Evaluations of Education Programs. Journal of Educational and Behavioral Statistics, vol. 33, no. 1. https://www.mathematica.org/publications/journal-article-statistical-power-for-random-assignment-evaluations-of-education-programs

q2 · i? · theoretical · r?

The article's abstract reports its central empirical conclusion: clustering effects are "typically large" across designs, and consequently "large school samples are required for education trials". No effect size or sample count is printed in the available text.

"Clustering effects vary by design but are typically large. As a result, large school samples are required for education trials, and many evaluations will have sufficient power to detect precise impacts only for relatively large subgroups of sites."

Peter Z. Schochet 2005¶

Peter Z. Schochet. (2005). Statistical Power for Random Assignment Evaluations of Education Programs. Princeton, NJ: Mathematica Policy Research. https://www.mathematica.org/publications/statistical-power-for-random-assignment-evaluations-of-education-programs

q2 · i? · theoretical · r?

The paper's empirical analysis of statistical power for education experiments finds that clustering effects are typically large across designs. The author reports that "large school samples are required" as a consequence, limiting the rigor of most impact studies to broad questions.

"Clustering effects vary by design but are typically large; consequently, large school samples are required, suggesting that most impact studies will only be able to address broad questions rigorously."

Discussion¶

Related Claims¶

• Large school samples are required to achieve appropriate precision standards in clustered education experiments — possibly the same claim (merge candidate)
• Sample size formulas for clustered designs with school- or teacher-level random assignment are derived using generalized estimating equation methods — related
• Most education impact studies can rigorously address only broad questions due to power constraints — related
• Many education evaluations have sufficient power to detect precise impacts only for relatively large subgroups of sites — related
• Required numbers of schools differ across school, classroom, and student random-assignment designs — a narrower finding that bears on this claim
• The evaluation used a school-level cluster-randomized quasi-experiment (N = 50) that was powered only to detect large effects (d = 0.81) — a narrower finding that bears on this claim
• In a large-scale 106-classroom cluster randomized trial, Building Blocks raised end-of-intervention REMA scores with an effect size of 0.48 (improvement index +18) — related
• In a 28-classroom cluster randomized trial, Building Blocks raised end-of-intervention REMA scores with an effect size of 1.07 (improvement index +36) — related
• The WWC Adolescent Literacy review rates PALS as having potentially positive effects on reading comprehension from one trial of students averaging 9.8 years old, with an effect size of 0.48 that is not statistically significant when adjusted for clustering — related
• Most studies of literacy resilience have (very) small sample sizes and low statistical power to find relevant effects — related