A substantial part of the LLM rating advantage appears attributable to comment length, as length-adjusted scores converge on every dimension except tone¶
Claim · All claims
Evidence · 1 study · associationalr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? After length adjustment, human ratings rise and LLM ratings fall on every dimension except tone, with the largest effects for specificity (1.85→0.45) and actionability (1.40→0.42). → Yijun Liu 2026
Evidence¶
Yijun Liu 2026¶
Yijun Liu, Yifan Song, John Gallagher, Sarah Sterman, Tal August. (2026). FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essays. https://arxiv.org/abs/2606.06271 doi:10.48550/arxiv.2606.06271
q2 · i? · associational · r2
Length-adjusted analysis (§5.1.1): OLS regression of each rubric dimension on word count for the 85 rated sentence-level items, with adjusted means evaluated at the overall mean length (≈64 words). The article reports "Adjustment convergesbothsides rather than just lowering LLM scores".
"We found that a substantial part of the apparent human-model rating gap appears to be driven by length. Adjustment convergesbothsides rather than just lowering LLM scores: at the average length, human ratings rise and LLM ratings fall on every dimension except tone."
Discussion¶
Related Claims¶
- LLM feedback receives higher expert ratings on six of seven quality dimensions, with tone the exception — related
- Human raters' preferences align with the LLM judge on TutorBench (Pearson r = 0.82 across metric-level win rates) — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
length-explains-llm-rating-advantage |
title |
A substantial part of the LLM rating advantage appears attributable to comment length, as length-adjusted scores converge on every dimension except tone |
description |
A substantial part of the LLM rating advantage appears attributable to comment length, as length-adjusted scores converge on every dimension except tone |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — yijun-liu-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.