Researchers should adopt strategies to avoid or mitigate small biases when powering studies for small impacts¶

Strategy · All strategies
Evidence · 1 claim (1 for) · 1 study (1 theoretical), q2 · 0 of 1 report an effect size · 1 claim rests on one study

Description¶

The paper's second stated purpose is to "recommend strategies researchers can use to avoid or mitigate these biases" — the small biases that become consequential as studies are powered to detect smaller impacts. For RCTs this means achieving much lower missing-data rates or justifying that missing data are unrelated to outcomes; for RDDs it means weighing the larger sample costs of robust statistical methods against their protection against incorrect regression models.

Design Implications¶

Context¶

Requirements¶

• Researchers designing studies powered to detect impacts below 0.20 standard deviations must plan for bias control at the design stage

Constraints¶

• The recommended strategies are specific to RCTs and RDDs, the two designs examined

Target Learners¶

• Education researchers and evaluators

Target Learning Goals¶

• Designing education impact evaluations that avoid false inferences from small biases

Claims¶

• Robust RDD methods protect against model misspecification but require much larger sample sizes to detect small effects [+W]

Related Strategies¶

• Asymdystopia Small Biases False Inferences

Examples¶

-

Key Sources¶

• John Deke, Thomas Wei, Tim Kautz. (2017). Asymdystopia: The Threat of Small Biases in Evaluations of Education Interventions that Need to be Powered to Detect Small Impacts. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://www.mathematica.org/publications/asymdystopia-the-threat-of-small-biases-in-evaluations-of-education-interventions-that-need