A framework that balances Type I and Type II errors for addressing the multiple testing problem in social policy impact evaluations¶

Theory · All theories
Evidence · no claims cited

Description¶

The article presents a framework for handling the multiple testing problem in social policy impact evaluations. Its stated purpose is to "balance[] Types I and II errors" so that evaluations controlling spurious findings do not simultaneously miss genuine impacts. The framework is positioned as an approach for evaluators who must test many outcomes and subgroup impacts and need a principled trade-off between falsely declaring effects and failing to detect real ones.

Design Implications¶

Context¶

Requirements¶

• Applies to evaluations that conduct hypothesis tests across multiple outcomes and subgroups, as the article frames the problem

Constraints¶

Target Learners¶

• Evaluators and researchers designing social policy and education impact evaluations

Target Learning Objectives¶

• Sound causal inference about program impacts when testing multiple outcomes and subgroups

Claims¶

Related Theories¶

• Bayesian hypothesis testing as an approach to the multiple testing problem
• Multi-armed RCT design-based framework for examining multiple interventions in a single study

Examples¶

-

Key Sources¶

• Peter Z. Schochet. (2009). An Approach for Addressing the Multiple Testing Problem in Social Policy Impact Evaluations. Education Review, vol. 33, no. 6. https://www.mathematica.org/publications/an-approach-for-addressing-the-multiple-testing-problem-in-social-policy-impact-evaluations