Grammatical Accuracy is the most stable and objective scoring dimension for LLMs, while Task Response is the most challenging¶
Claim · All claims
Evidence · 1 study · associationalr1·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Grammatical Accuracy yielded the highest correlations (r = 0.45-0.63) and lowest MAEs (0.67-0.94), which the authors attribute to grammatical features being more rule-based; Task Response correlations ranged from non-significant (Claude, r = 0.22) to strong (Llama, r = 0.61). → Tongxi Liu 2026
Evidence¶
Tongxi Liu 2026¶
Tongxi Liu, Luyao Ye, Wei Yan. (2026). A framework for evaluation of large language models in essay assessment: Reliability, alignment, and causal reasoning. Computers and Education: Artificial Intelligence. https://doi.org/10.1016/j.caeai.2026.100565
q2 · i? · associational · r1
Dimension-level alignment analysis across five LLMs on 60 essays, reported in Results §4.3. The article prints the correlation and MAE ranges for Grammatical Accuracy and separately reports Task Response as the most challenging criterion.
"Grammatical Accuracy emerged as the most stable dimension, with correlations of r = 0.45-0.63 and lower MAEs (0.67-0.94; Fig. 3)."
Discussion¶
Related Claims¶
- The automated scoring system's agreement with human raters varied widely by dimension, with information fidelity showing a weak, non-significant correlation — a broader claim this one bears on
- LLM choice significantly influences human-AI coding correspondence, with Claude Sonnet 4 performing best and GPT 4.1 Mini worst — related
- Combining the revised rubric with rationale-first evaluation yields the highest human–machine alignment, with dimension-specific gains — related
- Llama 4 Maverick shows the strongest human-LLM alignment, while Claude Sonnet 4 performs weakest with poor calibration — related
- In one study of repeated LLM scoring of 60 IELTS essays, Gemini 2.5 Flash was moderately reliable (ICC 0.63-0.75) while Claude Sonnet 4 and DeepSeek R1 stayed below 0.70 in most dimensions — related
- Causal discovery analysis finds four of five LLMs prioritize lexical features in scoring, while DeepSeek R1 uniquely prioritizes syntactic complexity — related
- LLM scoring performance is dimension-specific: Reasoning shows the strongest agreement and Terminology the weakest across all three strategies — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
grammatical-accuracy-most-stable-llm-scoring-dimension |
title |
Grammatical Accuracy is the most stable and objective scoring dimension for LLMs, while Task Response is the most challenging |
description |
Grammatical Accuracy is the most stable and objective scoring dimension for LLMs, while Task Response is the most challenging |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-12 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — tongxi-liu-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.