Small language models used as automated judges exhibit severe leniency bias against tutoring responses¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Phi4:14b scored gemma3:27b (Trial 1) responses at 91–96% across multiple configurations while a human rater scored the same responses at 61%. → H. Chad Lane 2026

Evidence¶

H. Chad Lane 2026¶

H. Chad Lane, Bryson Kageler. (2026). CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming. SLM4ED'26: The 1st Workshop of Small Language Models for Education. https://github.com/InviteInstitute/CSTutorBench doi:10.1145/3770761.3777333

q2 · i? · design · r2

Judge-selection analysis during pipeline development. The authors selected Claude Sonnet 4 instead, whose v4 rubric produced overall scores within 0–7 percentage points of the human baseline for three of four validated models, with deepseek-r1:8b showing a larger gap. No effect sizes are printed.

"We initially explored using SLMs as automated judges but found severe leniency bias. Phi4:14b, for example, scored gemma3:27b (Trial 1) responses at 91–96% across multiple configurations, while a human rater scored the same responses at 61%."

Discussion¶

Related Claims¶

• Automated judging aligns strongly with three domain experts (r = 0.82 for role fidelity) but exhibits a conservative bias, scoring ethical deviation 0.08 points lower than humans — reports the opposite
• Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related
• Both fine-tuned models show regression toward the mean in score-wise bias, and error rises monotonically above score 3.0, with high scores hardest to predict — related
• RAG prompting shows systematic leniency, over-scoring relative to teacher mean scores in all four rubric dimensions (total bias = 1.787) — related
• LLM-as-a-judge ratings found only small, incremental differences between systems, with ceiling effects on several criteria — related
• Independent LLM evaluators agreed on agent rankings but differed markedly in leniency, with ceiling effects compressing between-agent discrimination — related
• In one study of five LLMs scoring 60 undergraduate essays on IELTS rubric dimensions, systematic underestimation of human scores was the dominant bias trend, with some models also compressing scores into a narrow range — reports the opposite
• No consistent relationship was observed between false-positive and false-negative rates and either the question or the LLM model — reports the opposite