Type M and Type S error framework (Gelman and Carlin) for characterizing inferential errors from statistical significance filtering¶

Theory · All theories
Evidence · no claims cited

Description¶

This commentary organizes the inferential mistakes examined by Simpson (2022) and Sims et al (2022) using the "'Type M' (magnitude) and 'Type S' (sign) errors described by Gelman and Carlin (2014)." Type M errors concern errors in magnitude and Type S errors concern errors in sign of estimated effects. The framework serves as the conceptual lens through which the commentary argues that filtering research findings using statistical significance produces substantial errors in education research.

Design Implications¶

Context¶

Requirements¶

• Applied to settings where research findings are filtered using statistical significance, as in education impact evaluation

Constraints¶

Target Learners¶

• education researchers
• methodologists
• decision makers using evidence

Target Learning Objectives¶

• understanding inferential errors in causal inference
• interpreting impact estimates appropriately

Claims¶

Related Theories¶

Examples¶

• Use the BASIE framework to interpret impact estimates from evaluations

Key Sources¶

• John Deke. (2023). A Strong Case for Rethinking Causal Inference. Journal of Research on Educational Effectiveness (online ahead of print). https://www.mathematica.org/publications/a-strong-case-for-rethinking-causal-inference