Several RECALL effect sizes in the Xie et al. (2018) meta-analysis could not be reproduced, and one reported value (−0.639) is inconsistent with the primary study's data¶

Claim · All claims
Evidence · 1 study (2 entries) · theoretical r3 · q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Recalculated SMDs for Kuehl et al. (2014a) RECALL discrepant from the meta-analysis's reported values could only be recreated by combining RECALL with PICTORIAL MATCHING outcomes. → Sophia C. Weissgerber 2021 q2 i? The reported RECALL effect size of −0.639 for Eitel et al. (2014) text-only conditions could not be reproduced in multiple attempts; the primary data indicate an effect size of about −0.14. → Sophia C. Weissgerber 2021 (2)

Evidence¶

Sophia C. Weissgerber 2021¶

Sophia C. Weissgerber, Matthias Brunmair & Ralf Rummer. (2021). Null and Void? Errors in Meta-analysis on Perceptual Disfluency and Recommendations to Improve Meta-analytical Reproducibility. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09579-1

q2 · i? · theoretical · r3

Recalculation of standardized mean differences from the primary studies' reported means and standard errors yielded discrepant values; the reported values "could only be recreated when combining RECALL outcomes with PICTORIAL MATCHING outcomes" for each (sub-)group.

"when using the same approach and formula to calculate the SMD for the two (sub-)groups in Kuehl et al. (2014a) for RECALL, we surprisingly obtained discrepant values as those reported by Xie and colleagues in Figure 2"

Sophia C. Weissgerber 2021 (2)¶

Sophia C. Weissgerber, Matthias Brunmair & Ralf Rummer. (2021). Null and Void? Errors in Meta-analysis on Perceptual Disfluency and Recommendations to Improve Meta-analytical Reproducibility. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09579-1

q2 · i? · theoretical · r3

For Eitel et al. (2014) text-only conditions, the commentary reports the original values "indicate merely a small negative disfluency effect" and that applying Xie et al.'s approach "the effect size should be −0.14".

"The discrepancy could not be resolved in multiple attempts, so we do not know how Xie et al. (2018) obtained the effect size of −0.639, which they report in Figure 2 for RECALL."

Discussion¶

Related Claims¶

• The Xie et al. (2018) disfluency meta-analysis contained objective coding errors that reversed the reported direction of two primary studies' effects — related
• Misclassification of moderating variables in the Xie et al. (2018) meta-analysis decreases the trustworthiness of its moderator analysis — related
• Applying the reproducibility checks to TRANSFER performance, the Xie et al. (2018) results are (rather) robust, but RECALL and moderator results are doubtful — possibly the same claim (merge candidate)