Platform-generated scores and rubric-based teaching quality produced substantially divergent rankings of the same AI teaching agents¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Agent rankings by platform score and by 8-dimension rubric quality diverged: A8 ranked third by platform but last by rubric, and A6 rose from fourth to first. → Zhang Hui 2026
Evidence¶
Zhang Hui 2026¶
Zhang Hui, Qu Lihui, Zheng Jianmin, Xiong Yi, Bai Hongbo, Ji Ruiying, Liu Guohui, Chen Wanling, Cheng Zirui, Chen Youbang, Yang Chun-tao. (2026). What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education. JMIR Medical Education. https://doi.org/10.2196/96819
q2 · i? · design · r2
Rank comparison of 8 agents from 167 dialogues on the Chaoxing platform versus rubric-based scoring; the article reports "A8 ranked third by platform score but last by rubric quality, and A6 rose from fourth to first."
"Agent rankings from platform scores and from the 8-dimension rubric diverged substantially (Figure 6A). Two agents illustrate this most clearly: A8 ranked third by platform score but last by rubric quality, and A6 rose from fourth to first."
Discussion¶
Related Claims¶
- Strengthening an agent's empathic persona did not translate into broader knowledge delivery: the revised family-member agent had high role-play quality but the lowest knowledge coverage — related
- Independent LLM evaluators agreed on agent rankings but differed markedly in leniency, with ceiling effects compressing between-agent discrimination — related
- LLM-based rubric scores agreed with expert ratings at the total-score level and on cognitive-process dimensions, but agreement was low for more subjective dimensions — related
- AI teaching agents differed most on knowledge-related dimensions and least on role-play quality, with adaptive difficulty calibration a shared weakness — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
platform-rubric-rankings-diverge-ai-teaching-agents |
title |
Platform-generated scores and rubric-based teaching quality produced substantially divergent rankings of the same AI teaching agents |
description |
Platform-generated scores and rubric-based teaching quality produced substantially divergent rankings of the same AI teaching agents |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-12 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — zhang-hui-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.