Newer LLM generations do not consistently agree better with human essay graders¶
Claim · All claims
Evidence · 1 study · associationalr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Improvements in general language modeling capability do not necessarily translate into better agreement with human essay grading across datasets; GPT-4 achieved the lowest MAE on DREsS while GPT-5 performed better on ASAP. → Jerin George Mathew 2026
Evidence¶
Jerin George Mathew 2026¶
Jerin George Mathew, Sumayya Taher, Anindita Kundu, Denilson Barbosa. (2026). LLMs Do Not Grade Essays Like Humans. https://arxiv.org/abs/2603.23714 doi:10.48550/arxiv.2603.23714
q2 · i? · associational · r2
Comparison of successive GPT and Llama versions across the three datasets: the GPT family showed an overall MAE reduction from GPT-3.5 to GPT-5, but intermediate versions did not follow the trend, and Llama 4 showed lower agreement than earlier versions on ASAP but higher on DREsS.
"Improvements in general language modeling capability do not necessarily translate into better agreement with human essay grading across different datasets."
Discussion¶
Related Claims¶
- Claude3 and PaLM2 show substantial agreement but fall short on identifying failing grades — related
- Out-of-the-box LLMs agree with human essay grades far less than human raters agree with each other — related
- Few-shot prompting modestly improves LLM-human score agreement, inconsistently across datasets and models — related
- LLM agreement with human examiners is limited in fast mode and improves in reasoning or thinking modes — related
- Newer LLM generations such as GPT-4 generally outperform GPT-3.5 in short-answer grading accuracy, but this advantage is not consistent across all tasks and domains — reports the opposite
- LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
no-consistent-generational-improvement-llm-grading |
title |
Newer LLM generations do not consistently agree better with human essay graders |
description |
Newer LLM generations do not consistently agree better with human essay graders |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-12 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — jerin-george-mathew-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.