Statistical significance as the criterion distinguishing a program effect from chance¶

Theory · All theories
Evidence · 6 claims (6 for) · 5 studies (4 causal, 1 review), q2–q3 · 1 of 5 report an effect size · 6 claims rest on one study

Description¶

The brief's organizing account is that an observed difference between program and comparison groups counts as a program effect only when it is "not likely due to chance alone (that is, it is statistically significant)." This framework underlies the brief's guidance: a study finding no effect is one where the group difference did not meet this significance criterion, and interpreting such a result requires asking what could have caused it.

Design Implications¶

Context¶

Requirements¶

• Outcome data for both a program group and a group that did not receive the program, so a difference can be tested

Constraints¶

• Framed for education program evaluations read by school and district decision-makers; the excerpt does not elaborate thresholds or test procedures

Target Learners¶

• school and district decision-makers

Target Learning Objectives¶

• interpreting evaluation findings, including studies that find no effects

Claims¶

• Author-reported statistically significant program effects did not remain significant after WWC corrections for clustering and multiple comparisons [+W]
• Head Start shows no discernible effects on mathematics achievement for 4-year-old children [+W]
• BCIRC has potentially positive effects on English language development (effect size 0.29, not statistically significant) [+W]
• Polya and Dahmus methods do not differ on the problem solution criterion [+W]
• In a quasi-experimental study, CIRC showed no statistically significant effect on comprehension outcomes after WWC clustering and multiple-comparison adjustments [+W]
• A larger percentage of BCIRC students met the bilingual education exit criterion in language (39% versus 21%), not statistically significant [+W]

Related Theories¶

Examples¶

-

Key Sources¶

• Neil Seftor. (2016). What Does It Mean When a Study Finds No Effects?. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee