LLM-based rubric scores agreed with expert ratings at the total-score level and on cognitive-process dimensions, but agreement was low for more subjective dimensions¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q2 i? AI and expert total scores correlated positively (r=0.61; ICC=0.51), with agreement strongest on cognitive-process dimensions and near-zero or negative ICC for role-play quality and the two 5-point dimensions. → Zhang Hui 2026

Evidence¶

Zhang Hui 2026¶

Zhang Hui, Qu Lihui, Zheng Jianmin, Xiong Yi, Bai Hongbo, Ji Ruiying, Liu Guohui, Chen Wanling, Cheng Zirui, Chen Youbang, Yang Chun-tao. (2026). What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education. JMIR Medical Education. https://doi.org/10.2196/96819

q2 · i? · design · r2

Validation rescoring of a stratified subset of 40 dialogues spanning all 8 agents by a blinded medical-education expert using the identical rubric; the article reports "r=0.61; ICC=0.51" at the total-score level.

"AI and expert ratings were closely aligned in magnitude (mean totals 79.1 vs 78.5) and were positively correlated at the total-score level (r=0.61; ICC=0.51)."

Discussion¶

Related Claims¶

• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• The automated scoring system's agreement with human raters varied widely by dimension, with information fidelity showing a weak, non-significant correlation — related
• LLM-as-a-Judge scores correlate with blinded human expert judgments, more strongly for Medical Narrative Generation than Story Direction — related
• AI-assigned total scores on a handwritten chemistry exam align strongly with TA-assigned totals (R2 = 0.91), while per-problem agreement is lower (R2 = 0.61–0.85) — related
• Platform-generated scores and rubric-based teaching quality produced substantially divergent rankings of the same AI teaching agents — related
• In Support & Evidence and Claim & Focus, teacher and AES scores agreed at the lowest and highest rubric points but diverged in the middle, with teachers more likely to score a 2 and the AES a 3
• AI teaching agents differed most on knowledge-related dimensions and least on role-play quality, with adaptive difficulty calibration a shared weakness — related
• Expert-in-the-loop revision substantially improves medical accuracy and educational quality of generated storylines — related
• Expert tutors' lesson-planning and practice-design leaderboards separated AI models, and expert-rated practice design correlated with fewer student answer disputes — related