Judge and crowd broadly agree on which criteria matter, but the judge's near-identical top scores make its ranking of strong systems essentially random¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Learner adaptation and content coverage fill eight of the crowd's ten preference reasons and six of the judge's ten recorded criteria, yet the top systems' judge scores are nearly identical. → Lin 2026

Evidence¶

Lin 2026¶

Lin, Yi-Cheng; Guo, Yu-Kai; Chen, Szu-Chi; Feng, Bo-Han; Hsu, Yun-Man; Hsieh, Hsiang; Lin, Yu-Jung; Wu, Yue-Ling; Dong, Jia-Kai; Cheng, An-Yu; Huang, Yu-Han; Ieong, Lok-Lam; Chen, Kuan-Yu; Tchouang, Ming-Douo; Sun, Shao-Hua; Lin, Che; Ding, Jian-Jiun; Lee, Hung-yi. (2026). Findings of the First Teaching Monster Challenge: A Benchmark of Pedagogical Content Knowledge in AI Agents. arXiv. https://arxiv.org/abs/2608.08852 doi:10.48550/arxiv.2608.08852

q2 · i? · design · r2

Prevalence-versus-salience analysis of the 320 preliminary videos comparing the judge's metric prevalence with the crowd's 145 written pairwise rationales. "The two broadly agree on what matters", but criteria the judge records on nearly every video, such as pure-calculation bias on 99.7% of videos, separate nothing.

"The two broadly agree on what matters, since learner adaptation and content coverage filleightofthecrowd'stenreasonsandsixofthejudge'stencriteria."

Discussion¶

Related Claims¶

• The LLM-judge separates weak submissions but its ranking of the strongest systems has almost no agreement with crowd ranking (Spearman ρ = −0.17) — related
• Within-judge variance of the repeated LLM-judge rubric is negligible, but cross-model-family agreement remains untested — related
• Current AI instructional-video systems score higher on content dimensions (Accuracy, Logic) than on delivery dimensions (Engagement, Adaptability), with Adaptability lowest — related
• Only the challenge winner clearly beat the organizer baselines; most shortlisted systems have not yet improved on a commercial generator or well-chosen human-made video — related
• Independent LLM evaluators agreed on agent rankings but differed markedly in leniency, with ceiling effects compressing between-agent discrimination — related
• LLM-as-a-judge ratings found only small, incremental differences between systems, with ceiling effects on several criteria — related
• Systems adapt to the specified learner persona in ways human raters can perceive, scoring well above chance in persona identification — related