Pair convenient platform analytics with transparent, education-specific evaluation criteria rather than trusting either alone¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The article recommends treating evaluation itself as a first-class research object: reporting what a metric measures and "pairing convenient platform analytics with transparent, education-specific criteria rather than trusting either alone." Because platform scores index student performance while rubrics index agent teaching behavior, the two answer different questions and should complement rather than replace each other. A pragmatic division of labor follows: automated scoring is trusted most for dimensions where it is demonstrably calibrated, while human judgment is reserved for contested, subjective dimensions.

Design Implications¶

Context¶

Requirements¶

• A transparent, multidimensional rubric validated against human expert judgment
• Human calibration of automated scoring, especially for subjective or affective dimensions

Constraints¶

• Automated and expert judgments aligned well on cognitive-process dimensions but poorly on more subjective, affective ones

Target Learners¶

• medical students
• educators adopting AI teaching agents

Target Learning Goals¶

• accurate evaluation of AI tutor teaching quality
• agent selection and refinement decisions

Affordances¶

• Eight Dimension Pedagogical Rubric Ai Teaching Agents

Related Strategies¶

• Equitable Assessment Rubrics
• Complement outcome measures with continuous process measures to assess social effects and enable adaptive pedagogical agents
• Strengthen IVR eye-tracking validity through theory-driven metric selection, transparent data-quality documentation, and triangulation with complementary methods
• Combine holistic with analytic scores as a complementary approach to enhance rating validity

Examples¶

-

Key Sources¶

• Zhang Hui, Qu Lihui, Zheng Jianmin, Xiong Yi, Bai Hongbo, Ji Ruiying, Liu Guohui, Chen Wanling, Cheng Zirui, Chen Youbang, Yang Chun-tao. (2026). What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education. JMIR Medical Education. https://doi.org/10.2196/96819