Use multiverse and specification-curve analysis to test meta-analytical robustness to subjective analytical choices¶

Strategy · All strategies
Evidence · 4 claims (4 for) · 1 study (1 theoretical), q2 · 0 of 1 report an effect size · 4 claims rest on one study

Description¶

For Analysis Reproducibility (Level 3), the article recommends adopting the multiverse and specification-curve approach for meta-analyses suggested by Voracek et al. (2019), identifying theoretically guided eligible specifications. The specification-curve puts the original authors' chosen analysis in perspective: the more the original effect sizes deviate from the distribution of alternative analytical models, the less robust and credible they are. If a specification curve is unfeasible, the article suggests a two-fold strategy of fitting similar plausible models and then multiple different but defensible ones on a sub-sample of the data.

Design Implications¶

Context¶

Requirements¶

• Relies on the same information sources as Level 2: data, code, and workflows including subjective decision logs; checks should be planned beforehand via a deposited reproducibility plan or preregistered protocol

Constraints¶

• Running all plausible models might prove unfeasible for meta-analyses with many outcome variables; the approach was only recently devised so almost all existing meta-analyses lack it

Target Learners¶

• meta-analysis authors
• reviewers
• replication researchers

Target Learning Goals¶

• assessing how sensitive meta-analytical effect sizes are to subjective analytical choices

Affordances¶

• Methods Reproducibility Four Levels Meta Analysis

Claims¶

• Applying the reproducibility checks to TRANSFER performance, the Xie et al. (2018) results are (rather) robust, but RECALL and moderator results are doubtful [+W]
• The Xie et al. (2018) disfluency meta-analysis contained objective coding errors that reversed the reported direction of two primary studies' effects [+W]
• Misclassification of moderating variables in the Xie et al. (2018) meta-analysis decreases the trustworthiness of its moderator analysis [+W]
• Several RECALL effect sizes in the Xie et al. (2018) meta-analysis could not be reproduced, and one reported value (−0.639) is inconsistent with the primary study's data [+W]

Related Strategies¶

Examples¶

-

Key Sources¶

• Sophia C. Weissgerber, Matthias Brunmair & Ralf Rummer. (2021). Null and Void? Errors in Meta-analysis on Perceptual Disfluency and Recommendations to Improve Meta-analytical Reproducibility. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09579-1