Neither LLM nor human coders showed systematic over- or under-estimation in extracting data from educational meta-analyses¶

Claim · All claims
Evidence · 1 study · design r3 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Bland-Altman plot inspection revealed no systematic bias for any human or LLM coder. → Jansen 2026

Evidence¶

Jansen 2026¶

Jansen, T., Liebenow, L. W., Schaller, N.-J., Hattie, J., & Möller, J. (2026). Automated Data Extraction by Large Language Models: Assessing Accuracy in Comparison to Human Experts Using the Example of Visible Learning. Educational Psychology Review. https://doi.org/10.1007/s10648-026-10136-5

q3 · i? · design · r3

Graphical Bland-Altman analysis of disagreements across all human and LLM coders in the 156-meta-analysis evaluation. The article reports "no evidence of systematic bias among any of the human or LLM coders"; no effect size is printed for this finding.

"A visual inspection of the disagreements using Bland-Altman plots (Supplement A) revealing no evidence of systematic bias among any of the human or LLM coders; that is, no coder consistently over- or under-estimated the extracted values."

Discussion¶

Related Claims¶

• Most extraction discrepancies stemmed from ambiguity within the source meta-analyses rather than coder fabrication — related
• LLM-based extraction of three variables from 156 meta-analyses cost about $50 in computing fees, far less than estimated human-assistant labor costs — related
• Hallucinations were rare for both human and LLM coders, whereas omission errors were more frequent and varied by coder — related
• Across 468 data points, human coders, LLMs, and both standards agreed on the same value in 280 cases (59.8%) — related