Out-of-the-box LLMs agree with human essay grades far less than human raters agree with each other¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across ASAP and DREsS datasets, LLM-human score agreement (QWK below 0.30 on ASAP; at most 0.34 on DREsS) is substantially weaker than human inter-rater agreement (QWK 0.72). → Jerin George Mathew 2026

Evidence¶

Jerin George Mathew 2026¶

Jerin George Mathew, Sumayya Taher, Anindita Kundu, Denilson Barbosa. (2026). LLMs Do Not Grade Essays Like Humans. https://arxiv.org/abs/2603.23714 doi:10.48550/arxiv.2603.23714

q2 · i? · associational · r2

Evaluation of six GPT and Llama models prompted out-of-the-box to score essays on ASAP Tasks 1 and 7 and DREsS, compared against human ratings using QWK, Pearson correlation, and normalized MAE. Human raters reached "QW K= 0.72" while all LLMs stayed below 0.30 QWK on ASAP.

"For the ASAP datasets, the agreement between the two human raters is relatively high (QW K= 0.72), indicating strong inter-rater reliability. In contrast, all LLMs achieve considerably lower agreement with human scores, with QWK values below 0.30 across models and tasks."

Discussion¶

Related Claims¶

• LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks — the general claim this finding bears on, with the studies behind it
• Newer LLM generations do not consistently agree better with human essay graders — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related
• Few-shot prompting modestly improves LLM-human score agreement, inconsistently across datasets and models — related
• Human inter-rater agreement on the writing rubric is imperfect: no rubric item reaches full agreement and the most discrepant rater generally falls below 80% — related
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• LLMs agree strongly with their own consensus grades (most ICC > 0.8), far more than with human teachers' grades — related
• Expert human raters established an inter-rater reliability benchmark above ICC 0.89 on all four IELTS dimensions for the essay corpus — related
• AEEGS scores show high absolute agreement with 12 human expert raters' scores on mathematical Economics essay items (average-measures ICC = 0.863) — related
• LLM-based scoring of Spanish argumentative writing achieves 60–80% per-item accuracy, slightly below human inter-rater agreement — related
• Human Graders of Open-Ended Work Agree With Each Other Less Than Treating One Grade as Ground Truth Implies — related
• Under single-pass deployment, Fs+CoT prompting yields the strongest agreement with teacher mean scores for music-analysis essay scoring (r = 0.795, ICC (2,1) = 0.657) — related
• LLM ratings of classroom transcripts are more correlated with each other than with expert human ratings, across the same and different tasks — a narrower finding that bears on this claim
• LLM Short-Answer Graders Agree With One Human Rater More, and the Other Less, Than the Two Raters Agree With Each Other — a broader claim this one bears on
• Claude3 and PaLM2 show substantial agreement but fall short on identifying failing grades — reports the opposite