State-of-the-art LLMs extract effect sizes, study counts, and participant counts from educational meta-analyses with excellent agreement with adjudicated gold and silver standards¶

Claim · All claims
Evidence · 1 study · design r3 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Across Gemini 2.5 Pro, GPT 4.1, and GPT o3, percentage agreement with the gold standard ranged from 77% to 81% and ICCs ranged from 0.96 to 0.97. → Jansen 2026

Evidence¶

Jansen 2026¶

Jansen, T., Liebenow, L. W., Schaller, N.-J., Hattie, J., & Möller, J. (2026). Automated Data Extraction by Large Language Models: Assessing Accuracy in Comparison to Human Experts Using the Example of Visible Learning. Educational Psychology Review. https://doi.org/10.1007/s10648-026-10136-5

q3 · i? · design · r3

Evaluation of three LLMs extracting three variables from 156 educational meta-analyses randomly sampled from Visible Learning, benchmarked against adjudicated standards. The article reports "percentage agreement ranged from 77% to 81%" and ICCs "ranging from 0.96 to 0.97" against the gold standard; against the silver standard agreements were 78% to 84% with ICCs 0.94 to 0.96. No effect size is printed for this finding.

"Across the three models, percentage agreement ranged from 77% to 81%, while the Intraclass Correlation Coefficients (ICC) were excellent, ranging from 0.96 to 0.97."

Discussion¶

Related Claims¶

• Most extraction discrepancies stemmed from ambiguity within the source meta-analyses rather than coder fabrication — related
• Hallucinations were rare for both human and LLM coders, whereas omission errors were more frequent and varied by coder — related
• LLM-based extraction of three variables from 156 meta-analyses cost about $50 in computing fees, far less than estimated human-assistant labor costs — related