In a pilot study, leading LLMs showed promising but imperfect agreement with human coders extracting data from K-12 virtual-character studies¶
Claim · All claims
Evidence · 1 study · designr2·q1
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? In the pilot study (eight documents, 24 variables), Claude 2.1 achieved 93.06% agreement on explicit variables and 80.00% on derived variables; GPT-4 achieved 91.67% and 64.17%; ChatPDF achieved 75% and 60%. → Schroeder 2025
Evidence¶
Schroeder 2025¶
Schroeder, N. L., Jaldi, C. D., Zhang, S., & Yamoah, K. (2025). AI-Assisted Data Extraction for Systematic Reviews in Education. https://osf.io/nraxf/?view_only=879654690be249cc9237525ed7f86ec2
q1 · i? · design · r2
Small pilot study in Spring 2024 using web-based chat interfaces with eight sources (four journal articles, four conference proceedings) and 24 variables. "Claude was the most accurate LLM, showing 93.06% agreement with the human coder". The authors state prompts were not optimized.
"Compared to the human coder, we found that Claude was the most accurate LLM, showing 93.06% agreement with the human coder. GPT-4 was also quite accurate (91.67% agreement), and ChatPDF was reasonably accurate (75% agreement)."
Discussion¶
Related Claims¶
- LLMs are substantially less accurate at extracting and categorizing derived data based on predefined categories than at extracting explicitly stated data — related
- LLMs show moderate agreement with human coders when extracting explicitly stated data from education studies — related
- LLMs are not ready to be used as primary data extraction tools without explicit human validation in education research synthesis — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
pilot-study-llm-agreement-virtual-character-studies |
title |
In a pilot study, leading LLMs showed promising but imperfect agreement with human coders extracting data from K-12 virtual-character studies |
description |
In a pilot study, leading LLMs showed promising but imperfect agreement with human coders extracting data from K-12 virtual-character studies |
status |
review |
evidence_strength |
weak |
newest_evidence |
2025 |
generated |
by process:wiki-ingest on 2026-10-28 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — schroeder-2025 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.