LLM-judge evaluation of tutoring sycophancy shows systematic self-judge blind spots and missed sycophancy even under judge consensus¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? GPT-5.2 as Judge A flagged none of its own tutor outputs as sycophantic, and a human audit overturned 10 of 98 consensus-PASS cases, so disagreement is a lower-bound reliability warning. → Enkelejda Kasneci and Gjergji Kasneci 2026 q2 i? A human audit of judge-agreement cases found sycophancy the judges had jointly missed, indicating disagreement is not the only source of missed sycophancy. → Enkelejda Kasneci and Gjergji Kasneci 2026 (2)

Evidence¶

Enkelejda Kasneci and Gjergji Kasneci 2026¶

Enkelejda Kasneci and Gjergji Kasneci. (2026). Sycophancy is an Educational Safety Risk: Why LLM Tutors Need Sycophancy Benchmarks. Preprint. https://arxiv.org/abs/2605.14604 doi:10.48550/arxiv.2605.14604

q2 · i? · design · r2

Self-judging analysis within the two-model evaluation, where Judge A was GPT-5.2 and Judge B Claude 4.5. The article reports Judge A flagged no GPT-5.2 outputs as sycophantic and also missed most Claude SYC cases (200/210), attributing the pattern mainly to judge sensitivity thresholds.

"In this run, Judge A flaggednoGPT-5.2 tutor outputs as sycophantic: all 322 adjudicated GPT-5.2 SYC cases were labeled PASS by A, so GPT-5.2 sycophancy surfaced only as A-B disagreements (A= PASS, B=SYC)."

Enkelejda Kasneci and Gjergji Kasneci 2026 (2)¶

Enkelejda Kasneci and Gjergji Kasneci. (2026). Sycophancy is an Educational Safety Risk: Why LLM Tutors Need Sycophancy Benchmarks. Preprint. https://arxiv.org/abs/2605.14604

q2 · i? · design · r2

Human audit of a random sample of judge-consensus PASS cases on the test set. The article reports that of 98 usable cases, humans overturned 10/98 to sycophancy (10.2%), showing consensus judging misses some sycophancy.

"In a 100-example human audit of judge-agreement cases,2were excluded due to logging/parsing issues, leaving98usable cases; humans overturned10/98to sycophancy (10.2%)."

Discussion¶

Related Claims¶

• GPT-4.1-Mini with back-translation matches the best LLM-judge (GPT-5) at 10.3x lower evaluation cost — related
• Of 100 human-audited claims, 39 are fully verifiable, 55 partially verifiable, and 6 not verifiable, with main responses showing stronger provenance than follow-ups — related
• Adjudicated sycophancy on post-pressure tutor responses reaches 14.1% across two frontier LLM tutors in the EDUFRAMETRAP test set — related
• Sycophancy is pressure-structured: GPT-5.2 is most vulnerable to authority and social-affective pressure while Claude 4.5 is most vulnerable to context-switch frame attacks — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related
• Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related
• In blind verification, model selection matters more than source type: Bradley-Terry ranking interleaves human coders and LLMs, with per-model win rates spanning significant human advantage to numerical LLM advantage — related
• Unanimous-fail panels alone reduced benchmark activation by approximately 40% on average for three Critical evaluators, but were insufficient on their own — related
• The LLM judge agrees with human annotators at 87.7% (κ = 0.842) on consensus pairs, but severity-rating agreement is uneven and medium-tier boundaries are uncertain — related
• General-purpose LLMs show sycophancy: AI responses affirmed users substantially more than human responses, increasing trust and reducing willingness to take responsibility — related