Across 468 data points, human coders, LLMs, and both standards agreed on the same value in 280 cases (59.8%)¶

Claim · All claims
Evidence · 1 study · design r3 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? The most frequent agreement pattern was full agreement among humans, LLMs, and both standards (280 of 468 comparison points; 59.8%). → Jansen 2026

Evidence¶

Jansen 2026¶

Jansen, T., Liebenow, L. W., Schaller, N.-J., Hattie, J., & Möller, J. (2026). Automated Data Extraction by Large Language Models: Assessing Accuracy in Comparison to Human Experts Using the Example of Visible Learning. Educational Psychology Review. https://doi.org/10.1007/s10648-026-10136-5

q3 · i? · design · r3

UpSet-plot analysis of agreement combinations across all coders and standards in the evaluation. The article reports "in 280 cases (59.8% of the 468 comparison points), human, LLM coder, and both standards agree"; other patterns included 57 cases of single-coder Visible Learning deviation (12.2%).

"The highest line on the left indicates that in 280 cases (59.8% of the 468 comparison points), human, LLM coder, and both standards agree."

Discussion¶

Related Claims¶

• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related
• Human-human agreement (average κ = 0.644) was lower than the best LLM-LLM agreement (average κ = 0.856) across constructs — related
• Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related
• Hallucinations were rare for both human and LLM coders, whereas omission errors were more frequent and varied by coder — related
• Neither LLM nor human coders showed systematic over- or under-estimation in extracting data from educational meta-analyses — related