LLMs agree strongly with their own consensus grades (most ICC > 0.8), far more than with human teachers' grades¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i? When the majority-vote grade across the 18 models was the reference, most models reached ICC values above 0.8 (highest gpt-4.1-mini at 0.892), while gpt-4.1-nano (0.599) and claude-haiku-3.5 (0.664) diverged most. → Marcin Jukiewicz 2025

Evidence¶

Marcin Jukiewicz 2025¶

Marcin Jukiewicz. (2025). A systematic comparison of Large Language Models for automated assignment assessment in programming education: Exploring the importance of architecture and vendor. arXiv preprint. https://arxiv.org/abs/2509.26483 doi:10.48550/arxiv.2509.26483

q2 · i? · associational · r2

Analysis using the most frequent grade across the 18 models as the reference (Table 6). The study reports "Most models reached ICC values above 0.8, indicating good agreement with the consensus", with the lowest values for gpt-4.1-nano (0.599) and claude-haiku-3.5 (0.664).

"The highest agreement with this consensus grade was obtained by gpt-4.1-mini (ICC = 0.892), followed closely by claude-opus-4.1 (0.888) and gpt-5-mini (0.877). Most models reached ICC values above 0.8, indicat- ing good agreement with the consensus."

Discussion¶

Related Claims¶

• Full-scale LLM versions consistently achieve higher agreement with human grades than their mini and nano counterparts — related
• Even the best LLM grader reaches only moderate agreement with human teachers, below the threshold for good reliability — related
• Out-of-the-box LLMs agree with human essay grades far less than human raters agree with each other — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related
• The cross-family LLM judge panel agrees with human domain experts on every judged axis, reaching the expert ceiling on answer-holding — related
• LLMs differ systematically in grading strictness, forming liberal, restrictive, and balanced grading styles on programming assignments — related
• Teachers systematically assign higher grades than all evaluated LLM grading models — related
• The LLM judge agrees with human annotators at 87.7% (κ = 0.842) on consensus pairs, but severity-rating agreement is uneven and medium-tier boundaries are uncertain