Multisite randomized trials as a design for estimating both average effects and cross-site variation¶

Theory · All theories
Evidence · 4 claims (4 for) · 2 studies (2 causal), q2–q4 · 0 of 2 report an effect size · 4 claims rest on one study

Description¶

The article presents multisite randomized controlled trials, in which "individuals are randomly assigned to alternative treatment arms within sites", as a design that can estimate both the cross-site average effect of treatment assignment (intent to treat) and the amount that impact varies across sites. It argues both statistics are substantively and methodologically important, but that only the average has been well studied, motivating its empirical contribution.

Design Implications¶

Context¶

Requirements¶

• Random assignment of individuals to treatment arms within sites, so that both average ITT effects and their cross-site variation can be estimated.

Constraints¶

Target Learners¶

• participants in education and workforce development programs

Target Learning Objectives¶

• estimating and understanding variation in program effects across implementation sites

Claims¶

• Cross Site Itt Effect Variation 16 Multisite Rcts [+M]
• Cross-site impact variation has implications for the statistical precision of multisite trials [+W]
• The study explores hypotheses about factors that predict the magnitude of cross-site impact variation [+W]
• BARR produces a small statistically significant positive effect on NWEA MAP math outcomes in a multi-site student-level randomized trial [+W]

Related Theories¶

• Design-based estimation framework extended from two-group RCTs to multi-armed RCTs

Examples¶

• Randomly assign students to treatment and control sections when course-level interventions can enroll large numbers

Key Sources¶

• Michael J. Weiss, Howard S. Bloom, Natalya Verbitsky-Savitz, Himani Gupta, Alma E. Vigil, Daniel N. Cullinan. (2017). How Much Do the Effects of Education and Training Programs Vary Across Sites? Evidence from Past Multisite Randomized Trials. Journal of Research on Educational Effectiveness, vol. 10, issue 4. https://www.mathematica.org/publications/how-much-do-the-effects-of-education-and-training-programs-vary-across-sites-evidence