Most extraction discrepancies stemmed from ambiguity within the source meta-analyses rather than coder fabrication¶

Claim · All claims
Evidence · 1 study · design r3 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Adjudication revealed that most errors came from ambiguous reporting in the meta-analyses, and all human extraction was judged plausible. → Jansen 2026

Evidence¶

Jansen 2026¶

Jansen, T., Liebenow, L. W., Schaller, N.-J., Hattie, J., & Möller, J. (2026). Automated Data Extraction by Large Language Models: Assessing Accuracy in Comparison to Human Experts Using the Example of Visible Learning. Educational Psychology Review. https://doi.org/10.1007/s10648-026-10136-5

q3 · i? · design · r3

Qualitative analysis of adjudicated disagreements in the 156-meta-analysis evaluation. The article reports "most errors stemmed from ambiguity within the source meta-analyses", citing complex reporting and the need to aggregate sample descriptions across multiple tables as common sources; no effect size is printed.

"Investigating disagreements in detail during our adjudication process revealed that most errors stemmed from ambiguity within the source meta-analyses, and all human extraction was plausible."

Discussion¶

Related Claims¶

• Hallucinations were rare for both human and LLM coders, whereas omission errors were more frequent and varied by coder — related
• Neither LLM nor human coders showed systematic over- or under-estimation in extracting data from educational meta-analyses — related
• State-of-the-art LLMs extract effect sizes, study counts, and participant counts from educational meta-analyses with excellent agreement with adjudicated gold and silver standards — related