Rank-based metrics can be misleading for analytic assessment: systems prompted for holistic scoring achieve strong, sometimes stronger, correlations with analytic scores than when prompted for those aspects¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? LLMs prompted for holistic scores show strong correlations with analytic fair average scores, in some cases stronger than when prompted for the respective analytic aspect, reflecting intercorrelations and a halo effect. → Stefano Bannò 2026

Evidence¶

Stefano Bannò 2026¶

Stefano Bannò, Kate Knill, Mark Gales. (2026). Towards Self-Referential Analytic Assessment: A Profile-Based Approach to L2 Writing Evaluation with LLMs. https://arxiv.org/abs/2605.04298 doi:10.48550/arxiv.2605.04298

q2 · i? · associational · r2

Correlational analysis in Section 3.3 (Table 3): e.g. GPT-4.1 prompted for holistic scores correlates 0.927 with Complexity vs 0.848 when prompted analytically; Qwen 2.5 correlates 0.754 with Willingness vs 0.457. The operational rater shows the same pattern.

"when we use LLMs prompted to predict holistic scores, we observe strong correlations with analytic scores, in some cases even stronger than when we prompt them for the respective analytic aspect."

Discussion¶

Related Claims¶

• Human operational raters remain stronger than LLMs at identifying relative strengths (positive feedback) within learner profiles — related
• In a self-referential framework, LLMs, particularly GPT-4.1, tend to outperform a single operational human rater at identifying relative weaknesses (negative feedback) across several Language and Content aspects — related
• Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related
• LLM-generated 7C collaboration assessment scores fall within the range of human expert variability across ten discussions — related
• A Gemini-based creativity autorater scores real students' complex multimedia creativity tasks on par with human experts (item Kappa 0.66; total-score Pearson r = 0.88) — related
• Ensembles of operational raters outperform the LLMs: two raters suffice on average across aspects, but larger ensembles are needed for individual aspects — related
• Holistic peer feedback correlated significantly with tutor marks while specific-criteria rubric feedback did not in the PEARS system — related