LLM feedback sentiment tracks assigned scores: low scores come with more criticism, high scores with more praise¶
Claim · All claims
Evidence · 1 study · associationalr2·q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Essays receiving low LLM scores are associated with high proportions of negative trait mentions, while higher-scored essays receive increasingly positive feedback, though the relationship varies in strength across traits and datasets. → Jerin George Mathew 2026
q2 i? In proxy XGBoost models predicting LLM scores from trait-specific praise and criticism counts, praise-related features produce positive SHAP values and criticism-related features negative ones. → Jerin George Mathew 2026
Evidence¶
Jerin George Mathew 2026¶
Jerin George Mathew, Sumayya Taher, Anindita Kundu, Denilson Barbosa. (2026). LLMs Do Not Grade Essays Like Humans. https://arxiv.org/abs/2603.23714 doi:10.48550/arxiv.2603.23714
q2 · i? · associational · r2
Aspect-based sentiment analysis (DeBERTa-v3 ABSA, confidence above 0.9) of trait-specific praise and criticism in LLM feedback, aggregated by assigned score for DREsS and ASAP Task 1 (Figures 11 and 12). Positive mentions frequently exceeded 80-100% for high-scoring essays on DREsS.
"essays receiving low scores are typically associated with a high proportion of negative mentions across traits, while essays receiving higher scores contain increasingly positive feedback."
Discussion¶
Related Claims¶
- Different LLMs rely on different subsets of rubric traits when assigning essay scores — related
- LLM feedback for students marked by race, language, disability, gender, or motivation shows positive feedback bias and feedback withholding bias — more praise and less substantive critique than for unmarked counterparts — related
- Motivation prompts shift feedback tone: unmotivated students receive enthusiastic, collaborative encouragement while motivated students receive direct, higher-level critique — related
- Gender prompts shift feedback stance: female students receive affective, first-person, empathy-oriented feedback while male students receive objective, task-oriented critique — related
- LLMs overrate short underdeveloped essays and over-penalize minor language errors in well-developed essays — related
- Both fine-tuned models show regression toward the mean in score-wise bias, and error rises monotonically above score 3.0, with high scores hardest to predict — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
llm-feedback-sentiment-tracks-scores |
aliases |
shap-praise-raises-criticism-lowers-llm-scores |
title |
LLM feedback sentiment tracks assigned scores: low scores come with more criticism, high scores with more praise |
description |
LLM feedback sentiment tracks assigned scores: low scores come with more criticism, high scores with more praise |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
source_checked |
2026-10-20 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-12 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — jerin-george-mathew-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.