Active learning algorithms reduce abstract-screening workload in education systematic reviews, saving on average 58% of irrelevant-record screening and 1.66 days at 95% sensitivity¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Using learning algorithms for abstract screening reduced the screening workload of irrelevant records by 58% on average (SD = 19%) and saved an estimated 1.66 days (SD = 1.80). → Campos 2024

Evidence¶

Campos 2024¶

Campos, D. G., Fütterer, T., Gfrörer, T., Lavelle-Hill, R., Murayama, K., König, L., Hecht, M., Zitzmann, S., & Scherer, R. (2024). Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research. Educational Psychology Review. https://doi.org/10.1007/s10648-024-09862-5

q2 · i? · causal · r2

Retrospective screening simulation of 27 education and educational psychology abstract collections in ASReview, evaluating ten learning algorithms across 270 runs. The article reports "a 58% (SD = 19%) reduction in the screening workload of irrelevant records" and 1.66 days saved on average.

"The results showed, on average, a 58% (SD = 19%) reduction in the screening workload of irrelevant records when using learning algorithms for abstract screening and an estimated time savings of 1.66 days (SD = 1.80)."

Discussion¶

Related Claims¶

• A data-driven stopping rule needs about 7% consecutive irrelevant abstracts to reach 95% sensitivity, saving on average 1.04 days — related
• The LR+SBERT model achieved the highest specificity and estimated time savings at 95% sensitivity among the tested learning algorithms — a narrower finding that bears on this claim
• Sensitivity under mixed stopping criteria correlated positively with collection length in some strategy groups but not others — related
• A time-based stopping rule requires screening about 70% of abstracts to reach 95% sensitivity, saving on average 0.86 days — related
• Model specificity correlates positively with database length and negatively with the proportion of relevant studies — related