Asymdystopia: small biases raise the risk of false inferences as studies are powered to detect smaller impacts¶

Theory · All theories
Evidence · no claims cited

Description¶

Asymdystopia is the authors' name for the phenomenon whereby "the potential for small biases to increase the risk of making false inferences as studies are powered to detect smaller impacts" grows. As education evaluations target effects below Cohen's "small" benchmark of 0.20 standard deviations, the same small bias can flip an inference, so the drive for finer detection creates a parallel need "to guard against smaller biases." The paper examines this for RCTs and RDDs.

Design Implications¶

Context¶

Requirements¶

• Studies powered to detect impacts smaller than 0.20 standard deviations must evaluate and control small sources of bias

Constraints¶

• The phenomenon is examined for randomized controlled trials and regression discontinuity designs, the two rigorous designs the paper analyzes

Target Learners¶

• Education researchers and evaluators designing impact studies

Target Learning Objectives¶

• Accurate inference about intervention effects in education research

Claims¶

Related Theories¶

Examples¶

• Researchers should adopt strategies to avoid or mitigate small biases when powering studies for small impacts

Key Sources¶

• John Deke, Thomas Wei, Tim Kautz. (2017). Asymdystopia: The Threat of Small Biases in Evaluations of Education Interventions that Need to be Powered to Detect Small Impacts. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://www.mathematica.org/publications/asymdystopia-the-threat-of-small-biases-in-evaluations-of-education-interventions-that-need