AI teaching agents differed most on knowledge-related dimensions and least on role-play quality, with adaptive difficulty calibration a shared weakness¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Between-agent variation was largest on knowledge-point coverage (CV=27.3%) and medical knowledge accuracy (CV=18.5%), smallest on role-play quality (CV=5.7%), and difficulty calibration was low across nearly all agents. → Zhang Hui 2026
Evidence¶
Zhang Hui 2026¶
Zhang Hui, Qu Lihui, Zheng Jianmin, Xiong Yi, Bai Hongbo, Ji Ruiying, Liu Guohui, Chen Wanling, Cheng Zirui, Chen Youbang, Yang Chun-tao. (2026). What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education. JMIR Medical Education. https://doi.org/10.2196/96819
q2 · i? · design · r2
Rubric scoring of 167 dialogues by Claude Opus 4.8 across 8 agents; agent totals ranged 60.6-91.6. The article reports coefficients of variation per dimension, with "role-play quality was the most uniform dimension (CV=5.7%)".
"It was largest on the 2 knowledge-related dimensions, namely knowledge-point coverage (CV=27.3%) and medical knowledge accuracy (CV=18.5%), indicating that agents differed most in what and how accurately they taught. In contrast, role-play quality was the most uniform dimension (CV=5.7%)"
Discussion¶
Related Claims¶
- Strengthening an agent's empathic persona did not translate into broader knowledge delivery: the revised family-member agent had high role-play quality but the lowest knowledge coverage — related
- Among 6,699 LLM-judge deduction flags, visual delivery is the largest deficiency category (39%) and learner adaptation the second (27%), while content problems are rarer (16%) — related
- Task category is the dominant factor in agent performance: cross-category spread (26.3 points) far exceeds cross-model spread (8.8 points) — related
- Platform-generated scores and rubric-based teaching quality produced substantially divergent rankings of the same AI teaching agents — related
- LLM-based rubric scores agreed with expert ratings at the total-score level and on cognitive-process dimensions, but agreement was low for more subjective dimensions — related
- Boundary compliance (S3) drives nearly all inter-model safety divergence, with a 53-point gap between the safest and least safe models — related
- Current AI instructional-video systems score higher on content dimensions (Accuracy, Logic) than on delivery dimensions (Engagement, Adaptability), with Adaptability lowest — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
agents-differ-knowledge-not-role-performance |
title |
AI teaching agents differed most on knowledge-related dimensions and least on role-play quality, with adaptive difficulty calibration a shared weakness |
description |
AI teaching agents differed most on knowledge-related dimensions and least on role-play quality, with adaptive difficulty calibration a shared weakness |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
source_checked |
2026-10-20 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-12 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — zhang-hui-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.