Author-reported statistically significant program effects did not remain significant after WWC corrections for clustering and multiple comparisons¶

Claim · All claims
Evidence · 1 study · review r? · q3
Currency · newest study 2006 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? In both studies, statistically significant positive findings reported by the study author were not statistically significant after WWC correction for clustering and multiple comparisons. → What Works Clearinghouse 2006

Evidence¶

What Works Clearinghouse 2006¶

What Works Clearinghouse, U.S. Department of Education. (2006). WWC Intervention Report: Too Good for Drugs. https://www.mendezfoundation.org

q3 · i? · review · r?

WWC footnote on its re-analysis of both randomized controlled trials. The WWC calculated statistical significance itself, correcting "for clustering and multiple comparisons" where necessary, and found no significant differences in either domain. No effect size is attached to this discrepancy note.

"In both studies reviewed for Too Good for Drugs™ Bacon reported statistically significant positive findings for several student outcomes, but after correcting for clustering and multiple comparisons the WWC found that the differences between the groups were not statistically significant."

Discussion¶

Related Claims¶

• Large Type M (magnitude) errors almost certainly exist when applying the statistical significance filter in education research — the general claim this finding bears on, with the studies behind it
• Too Good for Drugs has potentially positive effects on student behavior, with a WWC-computed domain average effect size of 0.25 that is not statistically significant — a narrower finding that bears on this claim
• At end of program, the largest Too Good for Drugs effect was on teacher-rated prosocial behaviors (effect size 0.52), not statistically significant after WWC correction — a narrower finding that bears on this claim
• Three of seven knowledge, attitudes, and values outcomes favor Facing History and Ourselves, including relationship maturity (best response) with effect size 0.45, but none remain statistically significant after WWC corrections — a narrower finding that bears on this claim
• Greenwood et al. (1993) reported a statistically significant CWPT effect on CTBS-Reading, but the WWC-corrected analysis was not statistically significant — a narrower finding that bears on this claim
• No individual alphabetics outcome reached statistical significance in the WWC analysis — related
• The WWC rated Waterford Early Reading Program™ as having potentially positive effects on alphabetics, based on a substantively important average effect size with no statistically significant individual outcomes — related
• In a five-year quasi-experimental study, the intervention program showed a statistically significant positive effect on fifth-grade reading performance assessment but not on the CTBS Reading subtest — related
• Statistically significant effects reported by the CDP studies were not statistically significant as recalculated by the WWC — a narrower finding that bears on this claim
• The DreamBox effect on the MAP Computation subtest was not statistically significant after WWC multiple-comparison correction — a narrower finding that bears on this claim
• The DreamBox effect on the MAP Measurement and Geometry subtest was not statistically significant after WWC multiple-comparison correction — a narrower finding that bears on this claim
• Nunnery and Ross (2007) reported significant grade 5 but not grade 8 TAAS effects, which were non-significant in WWC recalculation — a narrower finding that bears on this claim
• In a quasi-experimental study, CIRC showed no statistically significant effect on comprehension outcomes after WWC clustering and multiple-comparison adjustments — a narrower finding that bears on this claim
• Author-reported RD findings count toward effectiveness ratings only if standard errors account for clustering of students at unique forcing variable values — related
• A program is said to have an effect when the difference between program and non-program groups is statistically significant — a broader claim this one bears on
• In the Bramlett quasi-experiment, no comprehension subtest or ability-level subgroup showed a statistically significant CIRC effect after WWC adjustments — a narrower finding that bears on this claim
• DreamBox effects on the Problem Solving, Number Sense, and Statistics and Probability subtests were not statistically significant — related
• Everyday Mathematics® showed positive effects on TAAS math subtests (Concepts, Operations, Problem Solving) with effect sizes of 0.25 to 0.31 that were not statistically significant after clustering correction — a narrower finding that bears on this claim
• A comparison of 193 pre-registered and 193 non-preregistered studies found no significant differences in positive results, effect sizes, or error rates — related