Human raters' preferences align with the LLM judge on TutorBench (Pearson r = 0.82 across metric-level win rates)¶

Claim · All claims
Evidence · 1 study · design r2 · q2 · i3 large
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i3 In a blind pairwise study on 45 TutorBench sessions, human and LLM DeepTutor win rates are strongly correlated across the ten metric-level win-rate pairs (Pearson r = 0.82; Spearman ρ = 0.83). → Bingxi Zhao 2026

Evidence¶

Bingxi Zhao 2026¶

Bingxi Zhao, Jiahao Zhang, Xubin Ren, Zirui Guo, Tianzhe Chu, Yi Ma, Chao Huang. (2026). DeepTutor: Towards Agentic Personalized Tutoring. Technical Report, The University of Hong Kong. https://arxiv.org/abs/2604.26962 doi:10.48550/arxiv.2604.26962

q2 · i3 · design · r2

Blind pairwise human study on a domain-stratified subset of 45 TutorBench sessions (nine per domain), comparing anonymized DeepTutor and Naive Tutor outputs under identical profiles and tasks. The printed correlations r = 0.82 and ρ = 0.83 indicate strong human–judge alignment.

"human and LLMDeepTutorwin rates are strongly correlated across the ten metric-level win-rate pairs (Pearson𝑟= 0.82, 𝑝= 0.0038; Spearman𝜌= 0.83, 𝑝= 0.0027). This agreement suggests that the LLM judge is not merely favoringDeepTutorglobally"

Discussion¶

Related Claims¶

• DeepTutor's interactive tutoring gains are stable across five university-level domains, varying only 0.16 points in overall quality — related
• In blind verification, model selection matters more than source type: Bradley-Terry ranking interleaves human coders and LLMs, with per-model win rates spanning significant human advantage to numerical LLM advantage — related
• Blind expert verification shows no overall preference for human over LLM qualitative coding when sources are judged symmetrically — related
• The automated scoring system's agreement with human raters varied widely by dimension, with information fidelity showing a weak, non-significant correlation — related
• DeepTutor improves overall first-person interactive tutoring quality by 10.76% over a Naive Tutor baseline on TutorBench — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related
• A substantial part of the LLM rating advantage appears attributable to comment length, as length-adjusted scores converge on every dimension except tone — related
• LLM alignment with expert teaching ratings does not predict, and is often negatively associated with, alignment with student learning gains — reports the opposite
• LLM-as-a-Judge scores correlate with blinded human expert judgments, more strongly for Medical Narrative Generation than Story Direction — a broader claim this one bears on
• In the public TutorBench sample, agency-oriented rubric criteria are most concentrated in the active-learning use case (agency-dimension density 0.720, agency-skill density 0.411) — related
• Expert tutors' lesson-planning and practice-design leaderboards separated AI models, and expert-rated practice design correlated with fewer student answer disputes — related
• Automated metrics and LLM judges can misalign with expert human preferences in non-verifiable educational evaluation — reports the opposite
• The LLM-judge separates weak submissions but its ranking of the strongest systems has almost no agreement with crowd ranking (Spearman ρ = −0.17) — related
• An automated LLM assessor rates 3E clinical communication skills with agreement comparable to individual human raters — related
• In blind A/B comparison, raters prefer personalised (Adaptive) questions over non-personalised (Random-topic) questions about 68–69% of the time in Biology and Chemistry — related
• Solving-oriented and pedagogy-oriented composite scores on the public MathTutorBench leaderboard are only moderately correlated (Pearson 0.421, Spearman 0.690) — related
• Blinded board-certified clinicians prefer the RL-trained agent in 87.6% of side-by-side comparisons for overall clinical impression — related