Automated essay scoring can achieve high agreement with human raters but faces calibration constraints¶

Claim · All claims
Evidence · 1 study · design r? · q2
Currency · newest study 2019 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · overdue for a recheck since 2022: tool and model accuracy changes quickly

Subclaims¶

q2 i? AES agreement with human raters reached 94.6%–98.2% in one reviewed study, but the system requires large numbers of pre-scored assessments for calibration. → Zawacki-Richter 2019

Evidence¶

Zawacki-Richter 2019¶

Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? International Journal of Educational Technology in Higher Education. https://doi.org/10.1186/s41239-019-0171-0

q2 · i? · design · r?

The review reports this single study's agreement figures and notes the authors' caveat that AES is impractical in most small classrooms due to calibration requirements and may not suit all writing genres.

"Gierl, Latifi, Lai, Boulais, and Champlain's (2014) use of open source Java software LightSIDE to grade postgraduate medical student essays resulted in an agreement between the computer classification and human raters between 94.6% and 98.2%"

Discussion¶

Related Claims¶

• AEEGS scores show high absolute agreement with 12 human expert raters' scores on mathematical Economics essay items (average-measures ICC = 0.863) — related
• GPT-4 achieved almost perfect inter-rater reliability with faculty after three iterations of rubric engineering on three of four medical school OEQ questions — related
• In one reviewed study, GPT-4 automated essay scoring showed score inflation, with AI–human rating discrepancies exceeding 1.5 standard deviations in 38% of graduate-level bibliographies — reports the opposite
• The best AES models per FRISCO dimension reached strong agreement for Situation (QWK 0.728) and Clarity (QWK 0.763) but only fair agreement for Focus, Reason, and Inference in Indonesian physics essays — related
• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• Human inter-rater agreement on the writing rubric is imperfect: no rubric item reaches full agreement and the most discrepant rater generally falls below 80% — related
• The AES tool scored middle school essays more generously than teacher scorers overall and on three of four dimensions — a narrower finding that bears on this claim
• In classroom video labeling, high raw agreement (often exceeding 90%) co-occurred with low chance-corrected κ due to temporal misalignment — related
• LLM-based scoring of Spanish argumentative writing achieves 60–80% per-item accuracy, slightly below human inter-rater agreement — related