Evaluate AI standardized patient systems with both endpoint and process-sensitive outcomes¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The article recommends that "Future evaluations of LLM-based simulated patient systems should therefore include both endpoint and process-sensitive outcomes." Diagnostic correctness should be complemented by measures of communication quality, history-taking completeness, information elicitation, checklist coverage, and consultation structure, especially in early clinical training where students are still learning how to conduct the consultation. Process improvements should also be evaluated alongside safeguards against over-reliance and AI-induced never-skilling for novice learners.

Design Implications¶

Context¶

Requirements¶

• process measures such as communication quality, history-taking completeness, and checklist coverage alongside diagnostic endpoints

Constraints¶

• the study's own comparator was structured non-LLM materials, so human-feedback comparisons remain untested; no delayed follow-up was included, so durability of gains is unknown

Target Learners¶

• medical students in early clinical training

Target Learning Goals¶

• clinical interviewing performance
• communication quality
• diagnostic accuracy

Affordances¶

• Functional Complementarity Ai Human Clinical Education

Related Strategies¶

• Evaluate AI literacy interventions with behavioral measures of LLM interaction in addition to self-reports
• Frame genAI virtual patient experiences: prompt experimentation with AI capabilities and orient students to its limits

Examples¶

-

Key Sources¶

• Yang, L., Liu, H., Li, S., Jia, R., Xiao, Y., Chen, G., & Lu, L. (2026). Evaluating Scaffolding-Oriented Multi-Agent Large Language Model System for Clinical Interview Training. https://arxiv.org/abs/2609.10939. doi:10.48550/arxiv.2609.10939