Blinded board-certified clinicians prefer the RL-trained agent in 87.6% of side-by-side comparisons for overall clinical impression¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? In blinded side-by-side evaluations by board-certified clinicians (n=97), the ResidencyRL-trained agent was preferred in 87.6% of cases for overall clinical impression, with a 90.7% win rate for completeness of information gathering and 75.3% for management plan appropriateness. → Liévin 2026

Evidence¶

Liévin 2026¶

Liévin, V., Schmidgall, S., Strother, T., Bijamov, A., Goel, A., Palepu, A., Park, C., Balazadeh, V., Sun, M. W., Guerard, M., Chen, J., Steiner, D., Dhillon, V., Azar, I., Mehta, A., Spetsieris, N., Shah, S., Abdelrahim, M., Dahiya, A., Liu, Y., Chou, K., Matias, Y., Hassidim, A., Webster, D. R., Le, Q. V., Hadsell, R., Barral, J., Radebaugh, C., Faust, A., Azizi, S., Schaekermann, M., Chen, P.-H. C., Tu, T., Racz, D., & Yang, L. (2026). ResidencyRL: Reinforcement Learning in Simulated Clinical Environments. arXiv. https://arxiv.org/abs/2608.07418 doi:10.48550/arxiv.2608.07418

q2 · i? · causal · r2

Human expert evaluation in which 97 board-certified clinicians compared the ResidencyRL-trained agent against the base model in blinded side-by-side comparisons within an expert-optimized agentic harness. The article reports the trained agent "was preferred in 87.6% of cases for overall clinical impression" with named win rates for sub-dimensions. No standardized effect size is printed.

"in blinded side-by-side evaluations by board-certified clinicians (𝑛=97), the ResidencyRL-trained agent was preferred in 87.6% of cases for overall clinical impression, with significant advantages in completeness of information gathering (90.7% win rate) and management plan appropriateness (75.3% win rate)"

Discussion¶

Related Claims¶

• Multi-turn RL training in simulated clinical encounters improves diagnostic accuracy under adversarial conditions, raising rubric-based accuracy from 81.0% to 88.0% — related
• RL training reduces missed red flag rates by approximately one third under adversarial conditions, mitigating premature closure — related
• Procedural competencies from simulation-based RL training generalize to out-of-domain benchmarks including AMIE Mx, AgentClinic, and CRAFT-MD — related
• Human raters' preferences align with the LLM judge on TutorBench (Pearson r = 0.82 across metric-level win rates) — related