Expert human raters established an inter-rater reliability benchmark above ICC 0.89 on all four IELTS dimensions for the essay corpus¶

Claim · All claims
Evidence · 1 study · associational r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Three expert graders achieved excellent agreement after calibration discussions, with Task Response highest (ICC = 0.929) and Grammatical Accuracy lowest (ICC = 0.893). → Tongxi Liu 2026

Evidence¶

Tongxi Liu 2026¶

Tongxi Liu, Luyao Ye, Wei Yan. (2026). A framework for evaluation of large language models in essay assessment: Reliability, alignment, and causal reasoning. Computers and Education: Artificial Intelligence. https://doi.org/10.1016/j.caeai.2026.100565

q2 · i? · associational · r1

Baseline inter-rater reliability assessment using ICC(3,1) among three expert graders in educational psychology who independently scored all 60 essays with the IELTS rubric, then calibrated through iterative discussions, reported in Results §4.1.

"Task Response achieved the highest reliability (ICC = 0.929), followed closely by Coherence Cohesion (ICC = 0.910). Lexical Resource (ICC = 0.897) and Grammatical Accuracy (ICC = 0.893) also demonstrated excellent reliability according to established guidelines (Koo & Li, 2016)."

Discussion¶

Related Claims¶

• Raters differed significantly in severity/leniency in the MFRM analysis — the general claim this finding bears on, with the studies behind it
• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• Out-of-the-box LLMs agree with human essay grades far less than human raters agree with each other — related
• Scaled qualitative coding of preservice teachers' video-analysis responses reached very good interrater agreement — related
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• Human coding of other-oriented transacts in the debate corpus reached high inter-coder reliability (kappa 0.72 for reasoning, 0.7 for transactivity, 0.95 for self/other distinction) — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• Inter-rater agreement among pre-service teachers is low regardless of grading approach or essay quality — reports the opposite