Bayesian adaptive approach to randomized policy evaluations for estimating heterogeneous treatment effects¶

Theory · All theories
Evidence · 1 claim (1 for) · 1 study (1 theoretical), q2 · 0 of 1 report an effect size · 1 claim rests on one study

Description¶

The article presents a statistical method for randomized policy evaluations that efficiently estimates heterogeneous treatment effects, identifying what works for whom. It is designed to exploit high-volume, high-velocity data streams from public programs, including web-based tools, tablet-based educational curricula, and electronic health records. The method "enables evaluators to consider multiple candidate interventions simultaneously, matching each study subject with the intervention that is most likely to benefit him or her."

Design Implications¶

Context¶

Requirements¶

• Randomized policy evaluation trials with continuously accumulating data streams from program participants

Constraints¶

• The authors state the approach works efficiently only under the right circumstances, with benefits conditional on conditions identified through simulation

Target Learners¶

• Participants in public programs, including students in online courses and employment services participants

Target Learning Objectives¶

• Identifying which interventions benefit which subgroups of learners (heterogeneous treatment effects)

Claims¶

• Adaptive Allocation To Promising Arms [+M]

Related Theories¶

• Bayesian adaptive randomized program evaluation framework
• Bayesian hypothesis testing as an approach to the multiple testing problem

Examples¶

• Simulation study of Bayesian adaptive design conditions
• Match each study subject to the candidate intervention most likely to benefit them
• Use Bayesian adaptive designs to channel growing public-program data streams into evaluation

Key Sources¶

• Mariel McKenzie Finucane, Ignacio Martinez, Scott Cody. (2015). What Works for Whom? A Bayesian Approach to Channeling Big Data Streams for Policy Analysis. Working Paper 40, Mathematica Policy Research. https://www.mathematica.org/publications/what-works-for-whom-a-bayesian-approach-to-channeling-big-data-streams-for-policy-analysis