Unreliability in an outcome measure reduces the statistical power of treatment-control comparisons¶

Claim · All claims
Evidence · 1 study · theoretical r3 · q2
Currency · newest study 2009 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q2 i? An experiment with 80 percent power to detect a 0.20 standardized mean difference in true scores has power of only 71 percent if the outcome measure's reliability is 0.80. → May 2009

Evidence¶

May 2009¶

May, Henry, Irma Perez-Johnson, Joshua Haimson, Samina Sattar, and Phil Gleason (2009). Using State Tests in Education Experiments: A Discussion of the Issues (NCEE 2009-013). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://www.mathematica.org/publications/using-state-tests-in-education-experiments-a-discussion-of-the-issues

q2 · i? · theoretical · r3

Analytical discussion in the reliability section of the paper, drawing on classical true-score theory and prior power research, illustrating how unreliability inflates the minimum detectable effect. The paper states the adjusted MDE becomes 0.22 when reliability is 0.80, showing "power of 71 percent" in that case.

"an experiment with 80 percent power to detect a 0.20 standardized mean difference in true scores is reduced to having power of 71 percent if the actual reliability of the outcome measure is 0.80"

Discussion¶

Related Claims¶

• In clustered school-based RCTs, OLS mediator analyses yield precise teacher practice-achievement estimates only with about 150 to 200 study schools — related
• IF-AT-scored physics midterm and final exams yield good test reliability (α = 0.71 and 0.82; α50 = 0.86 and 0.90) — related
• Standardized test data proved too statistically unreliable for determining individual teacher effectiveness, and support for its use is waning — related
• Standardized effect sizes communicate the practical significance of results and enable meta-analyses and a-priori power analyses, facilitating cumulative science — related
• Mood-related measures (contentedness, smiling, fidgetiness) did not vary systematically across the reliable and unreliable conditions — related
• A low alpha for the dependent variable leaves the unstandardized regression coefficient unbiased but attenuates the standardized coefficient and reduces statistical power — possibly the same claim (merge candidate)
• Defining facilitation as speech produced by facilitators imposes a low performance ceiling on predictive facilitative models — a narrower finding that bears on this claim
• Comprehension strategy instruction shows a large overall effect, but reliability is undermined by few standardized measures — a narrower finding that bears on this claim
• Most studies of literacy resilience have (very) small sample sizes and low statistical power to find relevant effects — related
• Subjective symptoms are an unreliable measure of fatigue, and physiological tests have not yet yielded a reliable measurement method (Offner) — a narrower finding that bears on this claim