The LLM-judge separates weak submissions but its ranking of the strongest systems has almost no agreement with crowd ranking (Spearman ρ = −0.17)¶
Claim · All claims
Evidence · 1 study · designr3·q2·i1small
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i1 The judge's ranking of the ten shortlisted systems correlates negatively with the crowd's Elo ranking (ρ = −0.17); the eventual winner ranked fourth under the judge while the judge's top two finished eighth and tenth with the crowd. → Lin 2026
Evidence¶
Lin 2026¶
Lin, Yi-Cheng; Guo, Yu-Kai; Chen, Szu-Chi; Feng, Bo-Han; Hsu, Yun-Man; Hsieh, Hsiang; Lin, Yu-Jung; Wu, Yue-Ling; Dong, Jia-Kai; Cheng, An-Yu; Huang, Yu-Han; Ieong, Lok-Lam; Chen, Kuan-Yu; Tchouang, Ming-Douo; Sun, Shao-Hua; Lin, Che; Ding, Jian-Jiun; Lee, Hung-yi. (2026). Findings of the First Teaching Monster Challenge: A Benchmark of Pedagogical Content Knowledge in AI Agents. arXiv. https://arxiv.org/abs/2608.08852 doi:10.48550/arxiv.2608.08852
q2 · i1 · design · r3
Comparison of Layer 1 judge ranking against Layer 2 crowd Elo ranking for the ten shortlisted systems, showing "almostno agreement" (ρ = −0.17). The mismatch is not measurement noise: judge scores were stable across repeated runs (median within-cell SD 0.14), and tsunumon's 90% Elo interval overlapped neither Team 67's nor Kuro's.
"itsrankingofthesetenhasalmostno agreement with the Layer 2 crowd ranking (Spearmanρ= −0.17).tsunumon, the eventual winner, ranks only fourth under the judge, while the judge's top two,Team 67andKuro,finisheighthandtenthwiththecrowd."
Discussion¶
Related Claims¶
- Judge and crowd broadly agree on which criteria matter, but the judge's near-identical top scores make its ranking of strong systems essentially random — related
- Only the challenge winner clearly beat the organizer baselines; most shortlisted systems have not yet improved on a commercial generator or well-chosen human-made video — related
- Independent LLM evaluators agreed on agent rankings but differed markedly in leniency, with ceiling effects compressing between-agent discrimination — related
- Human raters' preferences align with the LLM judge on TutorBench (Pearson r = 0.82 across metric-level win rates) — related
- Within-judge variance of the repeated LLM-judge rubric is negligible, but cross-model-family agreement remains untested — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
llm-judge-ranks-strongest-systems-poorly |
title |
The LLM-judge separates weak submissions but its ranking of the strongest systems has almost no agreement with crowd ranking (Spearman ρ = −0.17) |
description |
The LLM-judge separates weak submissions but its ranking of the strongest systems has almost no agreement with crowd ranking (Spearman ρ = −0.17) |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — lin-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.