Sensitivity under mixed stopping criteria correlated positively with collection length in some strategy groups but not others¶

Claim · All claims
Evidence · 1 study · causal r2 · q2 · i3 large
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i3 Sensitivity under mixed stopping criteria correlated positively with the number of abstracts in groups A, D, and G (e.g., group A: ρ = 0.76), but not in groups B, C, E, F, H, and I. → Campos 2024

Evidence¶

Campos 2024¶

Campos, D. G., Fütterer, T., Gfrörer, T., Lavelle-Hill, R., Murayama, K., König, L., Hecht, M., Zitzmann, S., & Scherer, R. (2024). Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research. Educational Psychology Review. https://doi.org/10.1007/s10648-024-09862-5

q2 · i3 · causal · r2

Exploratory Spearman correlations of mixed-strategy sensitivity with dataset characteristics across the 27 collections. Group A showed "a positive correlation with the number of abstracts in an abstract collection" (ρ = 0.76, 95% CI [0.50, 0.89]).

"the sensitivity achieved under each mixed-based heuristic stopping criteria had a positive correlation with the number of abstracts in an abstract collection in groups A, D, and G (group A: ρ = 0.76, 95% CI [0.50, 0.89]"

Discussion¶

Related Claims¶

• A data-driven stopping rule needs about 7% consecutive irrelevant abstracts to reach 95% sensitivity, saving on average 1.04 days — related
• Model specificity correlates positively with database length and negatively with the proportion of relevant studies — related
• A time-based stopping rule requires screening about 70% of abstracts to reach 95% sensitivity, saving on average 0.86 days — related
• The LR+SBERT model achieved the highest specificity and estimated time savings at 95% sensitivity among the tested learning algorithms — related
• Active learning algorithms reduce abstract-screening workload in education systematic reviews, saving on average 58% of irrelevant-record screening and 1.66 days at 95% sensitivity — related