A replication approach for testing nonexperimental methods against experimental ITT impact estimates when experiments include substantial control crossover¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The paper develops a methodological strategy for evaluating nonexperimental methods by comparing their estimates to results from rigorous randomized experiments. As the abstract states, "This working paper develops a new replication approach that facilitates testing nonexperimental methods against rigorous experimental ITT impact estimates when the experiment includes substantial control crossover." The approach is designed for evaluation contexts where control-group members receive the treatment.

Design Implications¶

Context¶

Requirements¶

• An available rigorous randomized experiment whose ITT impact estimate can serve as the benchmark for replication

Constraints¶

• Designed for the context in which the experiment includes substantial control crossover

Target Learners¶

• education researchers and evaluation methodologists

Target Learning Goals¶

• improving the validity of causal impact estimates in education research

Related Strategies¶

• Use regression discontinuity designs to replicate or check experimental impact estimates for education interventions

Examples¶

-

Key Sources¶

• Brian Gill, Joshua Furgeson, Hanley S. Chiang, Bing-Ru Teh, Joshua Haimson, Natalya Verbitsky-Savitz. (2013). Replicating Experimental Impact Estimates with Nonexperimental Methods in the Context of Control Crossover. Working Paper 21. Cambridge, MA: Mathematica Policy Research. https://www.mathematica.org/publications/replicating-experimental-impact-estimates-with-nonexperimental-methods-in-the-context-of-control-crossover