LLMs are not ready to be used as primary data extraction tools without explicit human validation in education research synthesis¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2025 · checked against its sources 2026-10-29 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q2 i? Across all main-study analyses, LLMs frequently extracted inaccurate data (at least 7% and up to 39% of the time) and falsely claimed variables were not reported (3–8% of the time), leading the authors to conclude LLMs should not be relied on for fully automated extraction. → Schroeder 2025

Evidence¶

Schroeder 2025¶

Schroeder, N. L., Jaldi, C. D., Zhang, S., & Yamoah, K. (2025). AI-Assisted Data Extraction for Systematic Reviews in Education. https://osf.io/nraxf/?view_only=879654690be249cc9237525ed7f86ec2

q2 · i? · design · r2

Authors' conclusion from the main-study confusion-matrix analyses of 112 studies, where LLMs "had often extracted inaccurate information at least 7% of the time" and up to 39% of the time, and falsely reported variables as not reported 3–8% of the time.

"Our results show that in a research synthesis context in education, LLMs should not be relied on for fully automated data extraction tasks without human oversight and validation."

Discussion¶

Related Claims¶

• LLM data extraction accuracy varies by data type, with high precision but lower recall across tasks — related
• LLMs show moderate agreement with human coders when extracting explicitly stated data from education studies — related
• Hallucinations were rare for both human and LLM coders, whereas omission errors were more frequent and varied by coder — reports the opposite
• State-of-the-art LLMs extract effect sizes, study counts, and participant counts from educational meta-analyses with excellent agreement with adjudicated gold and silver standards — reports the opposite
• Neither LLM nor human coders showed systematic over- or under-estimation in extracting data from educational meta-analyses — related
• Most extraction discrepancies stemmed from ambiguity within the source meta-analyses rather than coder fabrication — related
• LLM-based extraction of three variables from 156 meta-analyses cost about $50 in computing fees, far less than estimated human-assistant labor costs — related
• In a pilot study, leading LLMs showed promising but imperfect agreement with human coders extracting data from K-12 virtual-character studies — related