Develop rigorous benchmarks for LLM data extraction tasks in education systematic reviews¶

Strategy · All strategies
Evidence · 5 claims (5 for) · 1 study (1 design), q2 · 0 of 1 report an effect size · 5 claims rest on one study

Description¶

The article identifies the absence of shared evaluation resources as a barrier: "at this time there are no existing benchmarks for data extraction tasks in systematic review contexts in education." The authors argue that developing rigorous benchmarks would help educational researchers select LLMs to assist data extraction, while cautioning that benchmarks are only indicators since creators could train models on public test items.

Design Implications¶

Context¶

Requirements¶

• Benchmarks should use non-public test items where possible to reduce contamination

Constraints¶

• Benchmarks are simply indicators; LLM creators could train their models on public test items to ensure strong performance

Target Learners¶

• Education researchers and tool developers in research synthesis

Target Learning Goals¶

• Selecting appropriate LLMs for data extraction assistance

Claims¶

• LLM data extraction accuracy varies by data type, with high precision but lower recall across tasks [+W]
• LLMs are substantially less accurate at extracting and categorizing derived data based on predefined categories than at extracting explicitly stated data [+W]
• LLMs show moderate agreement with human coders when extracting explicitly stated data from education studies [+W]
• LLMs are not ready to be used as primary data extraction tools without explicit human validation in education research synthesis [+W]
• In a pilot study, leading LLMs showed promising but imperfect agreement with human coders extracting data from K-12 virtual-character studies [+W]

Related Strategies¶

• Evaluate LLM data extraction through a staged meaningful-human-control sequence before integrating it into evidence synthesis
• Extend educational LLM benchmarking toward multilingual contexts, real classroom interaction data, and targeted training for identified capability gaps
• Rigorous evaluation roadmap for lecture-KG systems: gold-standard annotation, controlled ablations, and faithfulness metrics
• Prioritize prospective, in-curriculum evaluations of AI governance interventions over additional isolated benchmark studies
• Standardize reporting of LLM versions, processing dates, input selection criteria, and data quality protocols in future MCQ benchmarking research

Examples¶

-

Key Sources¶

• Schroeder, N. L., Jaldi, C. D., Zhang, S., & Yamoah, K. (2025). AI-Assisted Data Extraction for Systematic Reviews in Education. https://osf.io/nraxf/?view_only=879654690be249cc9237525ed7f86ec2