The cross-family LLM judge panel agrees with human domain experts on every judged axis, reaching the expert ceiling on answer-holding¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? A three-family LLM judge panel agrees with two blind domain experts on every judged axis (permutation p < 0.05), reaching the expert ceiling on answer-holding and tracking Helpfulness and the curriculum axes as a positive but more modest proxy. → Lee 2026

Evidence¶

Lee 2026¶

Lee, U.; Lee, S.; Jeong, Y.; Lee, E.; Shin, M.; Kwon, H. (2026). EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners. https://arxiv.org/abs/2608.03206 doi:10.48550/arxiv.2608.03206

q2 · i? · design · r2

Human validation with two domain experts scoring about 40 judged days per axis, blind to the panel. Panel–expert Spearman ρ was 0.33 (Helpfulness), 0.43 (Gagné), 0.49 (Rosenshine), and 0.82 (answer-holding), against expert–expert ceilings of 0.91, 0.64, 0.90, and 0.85.

"The panel agrees with the experts on every judged axis (permutation p <0.05), reaching the expert ceiling on answer-holding and tracking Helpfulness and the two curriculum axes as a positive but more modest proxy."

Discussion¶

Related Claims¶

• Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — reports the opposite
• Blind expert verification shows no overall preference for human over LLM qualitative coding when sources are judged symmetrically — related
• Agreement metrics and expert quality judgments diverge in both directions: human consensus can encode shared conservative bias that the verifier rejects in favor of LLM coding — related
• In blind verification, model selection matters more than source type: Bradley-Terry ranking interleaves human coders and LLMs, with per-model win rates spanning significant human advantage to numerical LLM advantage — related
• Back-translation evaluation outperforms LLM-as-a-Judge (code+image) in agreement with human raters across four models — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — reports the opposite
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• Automated judging aligns strongly with three domain experts (r = 0.82 for role fidelity) but exhibits a conservative bias, scoring ethical deviation 0.08 points lower than humans — related
• An open-source Llama3 model reached substantial agreement with a human coder when sentiment-labeling pre-extracted student reflection quotes — related
• Unanimous-fail panels alone reduced benchmark activation by approximately 40% on average for three Critical evaluators, but were insufficient on their own — related
• LLM-as-a-Judge scores correlate with blinded human expert judgments, more strongly for Medical Narrative Generation than Story Direction — related
• Automated metrics and LLM judges can misalign with expert human preferences in non-verifiable educational evaluation — reports the opposite
• LLMs agree strongly with their own consensus grades (most ICC > 0.8), far more than with human teachers' grades — related
• The LLM judge agrees with human annotators at 87.7% (κ = 0.842) on consensus pairs, but severity-rating agreement is uneven and medium-tier boundaries are uncertain
• An automated LLM assessor rates 3E clinical communication skills with agreement comparable to individual human raters — related
• LLM-as-judge automated evaluation can achieve human-level agreement when carefully validated — a broader claim this one bears on