In one study of repeated LLM scoring of 60 IELTS essays, Gemini 2.5 Flash was moderately reliable (ICC 0.63-0.75) while Claude Sonnet 4 and DeepSeek R1 stayed below 0.70 in most dimensions¶

Claim · All claims
Evidence · 1 study · associational r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Reliability stratified by model: Gemini 2.5 Flash reached ICC = 0.63-0.75 with slightly compressed scoring ranges, while Claude Sonnet 4 and DeepSeek R1 remained below 0.70 in most dimensions. → Tongxi Liu 2026

Evidence¶

Tongxi Liu 2026¶

Tongxi Liu, Luyao Ye, Wei Yan. (2026). A framework for evaluation of large language models in essay assessment: Reliability, alignment, and causal reasoning. Computers and Education: Artificial Intelligence. https://doi.org/10.1016/j.caeai.2026.100565

q2 · i? · associational · r1

Reliability analysis (ICC(3,1)) of three repeated scoring runs per model on 60 essays, reported in Results §4.2 and Table 3. The article prints the ICC ranges and characterizes the lower-reliability models as showing "reduced reproducibility".

"Gemini 2.5 Flash achieved moderate reliability (ICC = 0.63-0.75), reflecting stable yet slightly compressed scoring ranges, while Claude Sonnet 4 and DeepSeek R1 remained below 0.70 in most dimensions, indicating reduced reproducibility."

Discussion¶

Related Claims¶

• Llama 4 Maverick shows the strongest human-LLM alignment, while Claude Sonnet 4 performs weakest with poor calibration — related
• Causal discovery analysis finds four of five LLMs prioritize lexical features in scoring, while DeepSeek R1 uniquely prioritizes syntactic complexity — related
• Grammatical Accuracy is the most stable and objective scoring dimension for LLMs, while Task Response is the most challenging — related