When reading an evaluation, check that the study compared a group that received the program to a group that did not¶

Strategy · All strategies
Evidence · 2 claims (2 for) · 4 studies (4 causal), q2–q4 · 1 of 4 report an effect size · 1 claim rests on one study

Description¶

This strategy tells readers of effectiveness research to look for a comparison between outcomes of a group that received a program and outcomes of a group that did not, since "research studies compare outcomes for a group that received a program to outcomes of a group that did not." Readers should then check whether any observed difference is statistically significant before concluding the program had an effect.

Design Implications¶

Context¶

Requirements¶

• A study design that reports outcomes separately for a program group and a group that did not receive the program

Constraints¶

• Applies to studies intended to measure program effectiveness; the brief does not describe designs lacking a comparison group

Target Learners¶

• school and district staff reading evaluation studies

Target Learning Goals¶

• judging whether an evaluation supports a claim that a program had an effect

Claims¶

• Civic Online Reasoning Instruction Improves Evaluation [+M]
• Reading Partners tutoring significantly improves reading comprehension for grade 2-4 struggling readers, with a total effect size of +0.10 [+W]

Related Strategies¶

Examples¶

-

Key Sources¶

• Neil Seftor. (2016). What Does It Mean When a Study Finds No Effects?. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee