Automated judging aligns strongly with three domain experts (r = 0.82 for role fidelity) but exhibits a conservative bias, scoring ethical deviation 0.08 points lower than humans¶

Claim · All claims
Evidence · 1 study · design r2 · q2 · i3 large
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i3 Pearson correlations between mean human scores and automated scores were r = 0.82 for RF, r = 0.78 for ED, and r = 0.75 for CS (all p < 0.001), with inter-annotator agreement of Fleiss' kappa = 0.71. → Saqib Shouqi 2026

Evidence¶

Saqib Shouqi 2026¶

Saqib Shouqi, Abdullah Nazly, Januki Wanniarachchi, Ravisha De Alwis. (2026). Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation. ADScAI Conference, University of Moratuwa, Sri Lanka. https://arxiv.org/abs/2608.03166 doi:10.48550/arxiv.2608.03166

q2 · i3 · design · r2

A human validation study had three domain experts independently score a stratified sample of 60 conversation turns (20 per persona) on RF, ED, and CS using the same rubrics as the automated judge; the printed correlation was "r = 0.82 for RF," with Fleiss' kappa = 0.71 among humans.

"Pearson correlation between mean human scores and automated scores was 𝑟= 0.82for RF ( 𝑝< 0.001),𝑟= 0.78for ED ( 𝑝< 0.001), and𝑟= 0.75for CS (𝑝< 0.001), indicating strong alignment."

Discussion¶

Related Claims¶

• Claude showed the highest alignment with human BREQ responses, and interview-containing prompts aligned better than baseline prompts — related
• Multi-strategy adversarial evaluation lowers RPLA robustness scores by 0.174–0.203 points relative to a single-strategy baseline across three personas — related
• RPLA failures are temporally distributed: ethical violations are rare in the first three turns and become significantly more frequent after turn 6 — related
• The automated scoring system's agreement with human raters varied widely by dimension, with information fidelity showing a weak, non-significant correlation — related
• LLM-as-judge evaluation scored CyberAGENTS highest on domain grounding (4.46) and agent role fidelity (4.35) — related
• Inter-annotator agreement was moderate (combined mean Fleiss's κ of 0.62 across 23 groups), yet most students reported having reached consensus — related
• Small language models used as automated judges exhibit severe leniency bias against tutoring responses — reports the opposite
• The cross-family LLM judge panel agrees with human domain experts on every judged axis, reaching the expert ceiling on answer-holding — related
• AI scoring shows a systematic conservative bias, under-grading writing so 15.3% (2024) and 16.5% (2025) of human-passing tests would fail under AI scoring — related
• In one reviewed study, GPT-4 automated essay scoring showed score inflation, with AI–human rating discrepancies exceeding 1.5 standard deviations in 38% of graduate-level bibliographies — related
• LLM-as-a-judge ratings found only small, incremental differences between systems, with ceiling effects on several criteria — related
• In one study of five LLMs scoring 60 undergraduate essays on IELTS rubric dimensions, systematic underestimation of human scores was the dominant bias trend, with some models also compressing scores into a narrow range — related
• The LLM judge agrees with human annotators at 87.7% (κ = 0.842) on consensus pairs, but severity-rating agreement is uneven and medium-tier boundaries are uncertain
• High agreement between two LLM screeners indicates consistency among outputs, not accuracy, because models may exhibit correlated failure patterns — related
• Five judges classifying 63 test units reached average inter-judge correlation of 0.542 ± 0.019 on the two scales — related