LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks¶

Claim · All claims
Evidence · 1 study · review r3 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? Across the reviewed studies, GenAI performed worse than humans in 11 cases and comparably in 12 cases for grading quality. → Marcin Jukiewicz and Michał Wyrwa 2026 q3 i? For shorter, more straightforward tasks GenAI performance tends to be similar to human graders, while longer, complex, or subjective assignments reveal a lack of nuance. → Marcin Jukiewicz and Michał Wyrwa 2026

Evidence¶

Marcin Jukiewicz and Michał Wyrwa 2026¶

Marcin Jukiewicz and Michał Wyrwa. (2026). Can ChatGPT Replace the Teacher in Assessment? A Review of Research on the Use of Large Language Models in Grading and Providing Feedback. Appl. Sci. 2026, 16, 680. https://doi.org/10.3390/app16020680

q3 · i? · review · r3

The review's synthesis of 42 empirical studies reports the tally of grading comparisons: "GenAI performed worse than humans in 11 cases" and was comparable in 12. No pooled effect size is printed, so no magnitude is reported.

"The analyzed studies yield mixed results when comparing GenAI and human assessment. GenAI performed worse than humans in 11 cases. In 12 instances, its effectiveness was comparable to human ratings."

Studies bearing on this claim¶

Can a language model score learners' written constructed responses as a trained human marker would, and on which tasks? This table gathers the studies behind this claim and the 16 narrower claims that bear on it: 12 distinct studies (5 design, 3 associational, 2 review, 1 causal, 1 qualitative), q1–q3, 2 of 12 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• ChatGPT's exam scores were within 10% of human grades 70% of the time in a study of AI-based grading — related
• LLM-generated feedback is more comprehensive in volume than human feedback but is sometimes too generic, abstract, or misaligned with the assigned grade — a narrower finding that bears on this claim
• LLM choice significantly influences human-AI coding correspondence, with Claude Sonnet 4 performing best and GPT 4.1 Mini worst — related
• A short strict-grader preamble knocks 14 of 17 open-weights LLM graders out of the graded band on a computer-vision exam, with three stopping grading outright — related
• Purpose-built chatbot effects on educational performance are mixed, with some studies showing no benefits or worse outcomes than comparison conditions — a broader claim this one bears on
• LLMs perform worse on complex, open-ended, or subjective assignments than on short, clearly formulated answers — related
• AI scoring shows a systematic conservative bias, under-grading writing so 15.3% (2024) and 16.5% (2025) of human-passing tests would fail under AI scoring — related
• Course Context Raised an LLM Grader's Agreement on Terse Answers and Lowered It on Long Ones — related
• In one study of five LLMs scoring 60 undergraduate essays on IELTS rubric dimensions, systematic underestimation of human scores was the dominant bias trend, with some models also compressing scores into a narrow range — related
• Higher-scoring students tend to be more reliable graders (except the very best), and students are least biased when grading peers with similar scores — related
• In a self-referential framework, LLMs, particularly GPT-4.1, tend to outperform a single operational human rater at identifying relative weaknesses (negative feedback) across several Language and Content aspects — related
• Newer LLM generations such as GPT-4 generally outperform GPT-3.5 in short-answer grading accuracy, but this advantage is not consistent across all tasks and domains — related
• ChatGPT-3.5 feedback on EAP argumentative essays aligns closely with human instructor scoring and identifies surface-level issues, but is less effective on rhetorical structure and contextual nuance — a narrower finding that bears on this claim
• LLM Short-Answer Graders Agree With One Human Rater More, and the Other Less, Than the Two Raters Agree With Each Other — related
• Sentence embedding models reach only moderate agreement and need human supervision for open-ended grading — a narrower finding that bears on this claim
• Newer LLM generations do not consistently agree better with human essay graders — related
• LLMs overrate short underdeveloped essays and over-penalize minor language errors in well-developed essays — related
• Claude3 and PaLM2 show substantial agreement but fall short on identifying failing grades — related
• Even the best LLM grader reaches only moderate agreement with human teachers, below the threshold for good reliability — related
• Pooling Repeated LLM Grades Does Not Improve Agreement on Short Answers — a narrower finding that bears on this claim