Misclassification of moderating variables in the Xie et al. (2018) meta-analysis decreases the trustworthiness of its moderator analysis¶

Claim · All claims
Evidence · 1 study · theoretical r3 · q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Weissgerber and Reinhard (2017) were wrongly categorized as having no test expectancy, and Weltman and Eakin (2014) were wrongly categorized as 'no image'. → Sophia C. Weissgerber 2021

Evidence¶

Sophia C. Weissgerber 2021¶

Sophia C. Weissgerber, Matthias Brunmair & Ralf Rummer. (2021). Null and Void? Errors in Meta-analysis on Perceptual Disfluency and Recommendations to Improve Meta-analytical Reproducibility. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09579-1

q2 · i? · theoretical · r3

Checking coding decisions against the primary studies' method sections revealed misclassifications: Weissgerber and Reinhard participants "definitely expected a performance test" at immediate posttest, and Weltman and Eakin's method section states both words and figures were used.

"Given that Weissgerber and Reinhard (2017) found no immediate disfluency effect but a delayed one, while Weltman and Eakin ( 2014) found a disfluency advantage, this misclassification is problematic and decreases the trustworthiness of the moderator analysis."

Discussion¶

Related Claims¶

• The Xie et al. (2018) disfluency meta-analysis contained objective coding errors that reversed the reported direction of two primary studies' effects — related
• Applying the reproducibility checks to TRANSFER performance, the Xie et al. (2018) results are (rather) robust, but RECALL and moderator results are doubtful — a broader claim this one bears on
• Several RECALL effect sizes in the Xie et al. (2018) meta-analysis could not be reproduced, and one reported value (−0.639) is inconsistent with the primary study's data — related