Human operational raters remain stronger than LLMs at identifying relative strengths (positive feedback) within learner profiles¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? The operational rater outperforms all models on most Language aspects (Intelligibility, Accuracy, Fluency) plus Comprehensibility and Purposefulness for positive feedback. → Stefano Bannò 2026

Evidence¶

Stefano Bannò 2026¶

Stefano Bannò, Kate Knill, Mark Gales. (2026). Towards Self-Referential Analytic Assessment: A Profile-Based Approach to L2 Writing Evaluation with LLMs. https://arxiv.org/abs/2605.04298 doi:10.48550/arxiv.2605.04298

q2 · i? · design · r2

Positive-feedback F0.5 results (Table 5) from the same zero-shot experiment on ICNALE GRA; the quote is from the Results section and names both sides of the comparison, with OR averaging 34.57 vs LLM averages of 31.96, 31.84, and 35.56.

"While LLMs continue to perform strongly onComplexity,Logicality, andSophisti- cation, the OR outperforms all models on most Languageaspects, includingIntelligibility,Accu- racy, andFluency, as well as onComprehensibility andPurposefulness."

Discussion¶

Related Claims¶

• In a self-referential framework, LLMs, particularly GPT-4.1, tend to outperform a single operational human rater at identifying relative weaknesses (negative feedback) across several Language and Content aspects — reports the opposite
• Ensembles of operational raters outperform the LLMs: two raters suffice on average across aspects, but larger ensembles are needed for individual aspects — reports the opposite
• Rank-based metrics can be misleading for analytic assessment: systems prompted for holistic scoring achieve strong, sometimes stronger, correlations with analytic scores than when prompted for those aspects — related