The guide argues that overfocusing on intervention selection alone risks a 'magic bullet' view because average positive outcomes do not guarantee local success¶

Claim · All claims
Evidence · 1 study · theoretical r2 · q1
Currency · newest study 2017 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q1 i? Selecting a program with positive average outcomes without needs assessment, local outcome analysis, and careful implementation can perpetuate a magic-bullet concept of improvement. → Hale 2017

Evidence¶

Hale 2017¶

Hale, S., Dunn, L., Filby, N., Rice, J., & Van Houten, L. (2017). Evidence-based improvement: A guide for states to strengthen their frameworks and supports aligned to the evidence requirements of ESSA. San Francisco: WestEd. http://www.wested.org/resources/evidence-based-improvement-essa-guide-for-states

q1 · i? · theoretical · r2

Authors' interpretive argument in the framework section, not a tested result: the guide states that "a program produces positive outcomes on average" without guaranteeing success "in every case," so selection must be preceded by needs assessment and followed by implementation and local outcome analysis.

"overfocusing on the decision itself can perpetuate a "magic bullet" concept of improvement: the fact that a program produces positive outcomes on average does not mean that it will do so in every case."

Studies bearing on this claim¶

Does choosing an innovation with good average evidence secure results locally, or do implementation and support decide them? This table gathers the studies behind this claim and the 5 narrower claims that bear on it: 6 distinct studies (2 theoretical, 2 qualitative, 1 design, 1 causal), q1–q3, 0 of 6 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• The article cautions that CL-structured group activities are only a potential part of the solution for ER motivation, not a guarantee — related
• The increased number and specificity of policy requirements challenged implementation flexibility, with some viewing the policy as one-size-fits-all — related
• Fraction Face-Off! shows potentially positive effects on general mathematics achievement, with a WWC-calculated effect size of 0.64 and improvement index of +24 on NAEP selected items — related
• Fraction Face-Off! shows potentially positive effects on number and operations outcomes, with a WWC-calculated average effect size of 0.89 and improvement index of +31 — a narrower finding that bears on this claim
• Holding a strong goal intention alone does not guarantee goal achievement — a narrower finding that bears on this claim
• Follow-up or post-implementation contact by change agents relates most strongly to adoption success; without it adoption falters — related
• Continuous improvement can fall short when it becomes a compliance-based checklist activity detached from purpose and equity — related
• Practitioners recommended replacing quiz-style comprehension checks with interactive follow-up activities and separate behavior tracking — related