Few-shot prompting modestly improves LLM-human score agreement, inconsistently across datasets and models¶

Claim · All claims
Evidence · 1 study · causal r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Prompt variants including a student-grade variant and a few-shot variant with two example essays produced modest, inconsistent improvements in agreement with human scores. → Jerin George Mathew 2026

Evidence¶

Jerin George Mathew 2026¶

Jerin George Mathew, Sumayya Taher, Anindita Kundu, Denilson Barbosa. (2026). LLMs Do Not Grade Essays Like Humans. https://arxiv.org/abs/2603.23714 doi:10.48550/arxiv.2603.23714

q2 · i? · causal · r1

Evaluation of two prompt variants beyond the base template: one specifying the student's grade level and one including two example essays with human-assigned scores. Improvements appeared across QWK, Pearson correlation, and MAE in some cases, particularly for few-shot, but not consistently.

"Prompt variants such as few-shot prompting can modestly improve LLM-human score agreement, but these gains are not consistent across datasets or models."

Discussion¶

Related Claims¶

• Out-of-the-box LLMs agree with human essay grades far less than human raters agree with each other — related
• Newer LLM generations do not consistently agree better with human essay graders — related
• Prompting with category information or few-shot examples yields limited improvement in distribution alignment, with zero-shot closest for most models — related
• In-context learning did not consistently improve LLM annotation reliability over zero-shot prompting for engagement coding — related
• Few-shot prompting does not uniformly improve LLM performance and can degrade subskill mastery estimates, most notably for phonology — related
• Claude's few-shot scoring behaved erratically across items: best few-shot model on Item 1 but by far the worst model on Item 2 — related