Task category is the dominant factor in agent performance: cross-category spread (26.3 points) far exceeds cross-model spread (8.8 points)¶

Claim · All claims
Evidence · 1 study · design r? · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Cross-category variation in mean score far exceeds cross-model variation, with Language & Creativity highest (76.9) and STEM Reasoning lowest (50.6). → Junjie Yu 2026

Evidence¶

Junjie Yu 2026¶

Junjie Yu, Pengrui Lu, Weiye Si, Hongliang Lu, Jiabao Wu, Kaiwen Tao, Kun Wang, Lingyu Yang, Qiran Zhang, Xiuting Guo, Xuanyu Wang, Yang Wang, Yanjie Wang, Yi Yang, Zijian Hu, Ziyi Yang, Zonghan Zhou, Binghao Qiang, Borui Zhang, Chenning Li, Enchang Zhang, Feifan Chen, Feng Jian, Fengyin Sun, Hao Qiu, Hao Zheng, Haoran Zhu, Hongyu Liu, Jianbin Deng, Jiaxin Song, Jiaying Chi, Jiayou Shi, Jie Fang, Jinghui Zhong, Jingyu Zhou, Jinze Li, Junfeng Yi, Junyan Yu, Junzhi Xue, Ni Song, Pengyi Chen, Qi Chen, Quansheng Li, Rui Tao, Shenghai Gong, Shenhang Lu, Tianqi Shen, Tianxiang Zhu, Tiehan Kang, Tingyu Li, Wendi Wu, Xiao Shen, Xiao Zhou, Xiaotao Zhang, Xinrong Li, Xuankun Yang, Xun Zhang, Yan Li, Ye Lu, Yi Wang, Yibo Zhou, Yichi Zhang, Yihao Sun, Yijun Huang, Yixin Zhu, Yixuan Wu, Yuchen Sun, Yue Wu, Yuheng Sun, Yukun Li, Yutian Tu, Yuxuan Qin, Yuzhuo Wu, Zeyu Li, Zhengyu Lou, Zhenning Ran, Zizhu He, Pengfei Liu. (2026). AcademiClaw: When Students Set Challenges for AI Agents. arXiv preprint. https://arxiv.org/abs/2605.02661 doi:10.48550/arxiv.2605.02661

q2 · i? · design · r?

Domain-level analysis of the six-model evaluation (Table 4 and Figure 4), reporting per-category mean scores across models. The quote's printed spreads show what a task tests matters more than which model attempts it; no standardized effect size is printed.

"The cross-model mean ranges from 76.9 (Language & Creativity) down to 50.6 (STEM Reasoning)—a 26.3-point gap—whereas the cross-category mean ranges only from 71.9 (Opus) to 63.1 (MiniMax)."

Discussion¶

Related Claims¶

• Risk category explains the largest share of variation in educational attack success, far more than LLM usage context or curriculum topic — related
• AI teaching agents differed most on knowledge-related dimensions and least on role-play quality, with adaptive difficulty calibration a shared weakness — related
• The knowledge base supported downstream expert-level analysis, yielding 1,267 risk items across 81 categories and 14 domains — related
• In the agent-based model, task difficulty and AI quality set the baseline level of overreliance, with difficulty the dominant global factor — related
• Across 17 frontier models, Stage 1 pedagogical-judgment scores are high (mean 0.893) while rank correlations with Stage 2 (ρ=0.24) and Stage 3 (ρ=0.21) are weak, with workflow scores topping out at 0.704 — related
• Model performance varies by task category, with Basic Geometry easiest and Spiral consistently the most challenging category — a narrower finding that bears on this claim
• Within General Capability, model spread concentrates in competition mathematics (AIME), which dominates the aggregate score — related