Only the challenge winner clearly beat the organizer baselines; most shortlisted systems have not yet improved on a commercial generator or well-chosen human-made video¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Under the Layer 2 crowd protocol, all three organizer baselines placed between first and second against the first-, second-, and fifth-ranked systems. → Lin 2026

Evidence¶

Lin 2026¶

Lin, Yi-Cheng; Guo, Yu-Kai; Chen, Szu-Chi; Feng, Bo-Han; Hsu, Yun-Man; Hsieh, Hsiang; Lin, Yu-Jung; Wu, Yue-Ling; Dong, Jia-Kai; Cheng, An-Yu; Huang, Yu-Han; Ieong, Lok-Lam; Chen, Kuan-Yu; Tchouang, Ming-Douo; Sun, Shao-Hua; Lin, Che; Ding, Jian-Jiun; Lee, Hung-yi. (2026). Findings of the First Teaching Monster Challenge: A Benchmark of Pedagogical Content Knowledge in AI Agents. arXiv. https://arxiv.org/abs/2608.08852 doi:10.48550/arxiv.2608.08852

q2 · i? · design · r2

Crowd evaluation of the cascaded pipeline, NotebookLM commercial generator, and static retrieval framework against ranked systems (Table 2, wins out of 9 comparisons); all three baselines placed in slots 1–2. The judge scored the static retrieval framework lowest, but its penalties came mostly from missing or partly covering the requested topic, not factual errors.

"Only the challenge winnertsunumonbeat the baselines, so the systems below it have not yet clearly improved on a mature commercial product or a well-chosen human-made video."

Discussion¶

Related Claims¶

• The LLM-judge separates weak submissions but its ranking of the strongest systems has almost no agreement with crowd ranking (Spearman ρ = −0.17) — related
• Judge and crowd broadly agree on which criteria matter, but the judge's near-identical top scores make its ranking of strong systems essentially random — related
• Systems adapt to the specified learner persona in ways human raters can perceive, scoring well above chance in persona identification — related
• Current AI instructional-video systems score higher on content dimensions (Accuracy, Logic) than on delivery dimensions (Engagement, Adaptability), with Adaptability lowest — related