In one reviewed study, GPT-4 automated essay scoring showed score inflation, with AI–human rating discrepancies exceeding 1.5 standard deviations in 38% of graduate-level bibliographies¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Automated essay scoring with GPT-4 inflated scores relative to human raters in a substantial share of graduate-level bibliographies. → Long DY 2026

Evidence¶

Long DY 2026¶

Long DY, Wang S, Md Rashid S and Lu XT. (2026). Artificial intelligence in higher education: a systematic review of its impact on student engagement and the mediating role of teaching methods. Front. Educ. 10:1648661. https://doi.org/10.3389/feduc.2025.1648661

q2 · i? · review · r2

The review reports, citing Manning et al. (2025), automated essay scoring experiments finding "GPT-4's tendency toward score inflation"; the 1.5 SD threshold and 38% rate are printed as reported.

"Automated essay scoring experiments also identified GPT-4's tendency toward score inflation, with AI–human rating discrepancies exceeding 1.5 standard deviations in 38% of graduate-level bibliographies ( Manning et al., 2025 )."

Discussion¶

Related Claims¶

• LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks — the general claim this finding bears on, with the studies behind it
• Implicit demographic conditioning inflates AES scores in some models, with Llama-70B scores rising 1.57 points over its own default, while GPT models remain stable — related
• In one community-college lab class, six scorers rating six teams with an engineering rubric differed by a mean of 1.2 of 9 points, which the authors read as a sign the rubric may not be reliable — related
• The fine-tuned BERT model's classification performance is consistent across stereotype-threatened and non-threatened students and other demographic subgroups — related
• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• Automated judging aligns strongly with three domain experts (r = 0.82 for role fidelity) but exhibits a conservative bias, scoring ethical deviation 0.08 points lower than humans — related
• Transformer-based NLP models achieve automated essay scoring with correlations to human grading exceeding 0.85 — related
• In one study of five LLMs scoring 60 undergraduate essays on IELTS rubric dimensions, systematic underestimation of human scores was the dominant bias trend, with some models also compressing scores into a narrow range — reports the opposite
• Automated essay scoring can achieve high agreement with human raters but faces calibration constraints — reports the opposite
• AI scoring shows a systematic conservative bias, under-grading writing so 15.3% (2024) and 16.5% (2025) of human-passing tests would fail under AI scoring — reports the opposite
• LLMs exhibit demographic sensitivity: systematic output differences attributable solely to demographic context when task inputs are held constant — related
• Teachers systematically assign higher grades than all evaluated LLM grading models — related
• Most faculty–GPT-4 scoring discrepancies were only one-point differences, shrinking across iterations — related
• The proposed framework outperforms fine-tuned BERT, Instructor-LM, GPT-4, and RAPT on TeacherEval-2023 across consistency, credibility, calibration, and fairness metrics — related
• LLMs use fewer extreme grades than human examiners, with grade-dependent opposing biases compressing grades toward the middle — related