Methods for estimating impacts when treatment effects vary apply to randomized controlled trials and are also applicable to quasi-experimental designs¶

Claim · All claims
Evidence · 1 study · review r? · q2
Currency · newest study 2014 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? The methods addressed focus on randomized controlled trials but extend to quasi-experimental designs. → Peter Z. Schochet 2014

Evidence¶

Peter Z. Schochet 2014¶

Peter Z. Schochet, Mike Puma, John Deke. (2014). Understanding Variation in Treatment Effects in Education Impact Evaluations: An Overview of Quantitative Methods. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee

q2 · i? · review · r?

The report states its scope: the quantitative methods for varying treatment effects center on RCTs while remaining "applicable to quasi-experimental designs". This is a scope statement rather than an empirical finding; no effect size is reported.

"The focus is on randomized controlled trials, but the methods are also applicable to quasi-experimental designs."

Discussion¶

Related Claims¶

• The intent-to-treat experimental impact estimate is the most causally rigorous measure for nonexperimental methods to replicate — related
• Quantitative methods exist for assessing how impacts of educational interventions on instructional practices and student learning differ across students, educators, and schools — a broader claim this one bears on
• Nonexperimental methods incorporating pre-treatment outcome measures can replicate experimental ITT impact estimates when control-group crossover occurs — related
• Control-group noncompliance makes experimental and nonexperimental estimands diverge (CACE vs. all treated subjects) — related
• Substantial crossover occurred in the Head Start impact study comparison group, with 17.3% of the 3-year-old and 13.9% of the 4-year-old comparison groups enrolling in non-study Head Start programs — related
• Replication of ITT estimates holds even when treatment effects differ for compliers and always-takers — related
• The Kauffman School evaluation used a matched comparison group design to estimate impacts on achievement, attendance, and suspensions — a narrower finding that bears on this claim
• Cross-site standard deviations of ITT effects were estimated for education and workforce interventions using data from 16 large multisite randomized controlled trials — a narrower finding that bears on this claim
• Not all intervention programs can be evaluated using an RCT, so non-experimental study designs may be needed to build an evidence base — related
• The study explores hypotheses about factors that predict the magnitude of cross-site impact variation — related
• Causal inference always goes beyond the data, even in randomized experiments, because experimental manipulations may affect other variables that caused the effect — related
• Quasi-experimental and experimental interventions can positively change epistemic cognition, but the theory-of-change remains unidentified — related