Related-case overlap between MedGame Bench train and test splits shows no significant performance difference¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Test cases with and without similar patient variants in the training split showed no significant performance difference (p > 0.05); equivalence was not tested. → Qian Wu 2026

Evidence¶

Qian Wu 2026¶

Qian Wu, Xinrong Zhou, Zizhan Ma, Kai Chen, Zheyao Gao, Xun Lin, Hongqiu Wu, Longfei Gou, Yixiao Liu, Ann Sin Nga Lau, Qi Dou. (2026). MedGame: Storytelling Gamification Empowered by Large Language Models for Medical Education. arXiv preprint. https://arxiv.org/abs/2607.21570 doi:10.48550/arxiv.2607.21570

q2 · i? · design · r2

Secondary analysis tracking high-similarity patient variants across the benchmark split (409 of 1,000 test cases had a similar variant in train). The article reports no significant difference and "no evidence of score inflation from related cases"; no effect size printed.

"Test cases with and without similar patient variants in the training split show no significant perfor- mance difference (p >0.05; Appendix E.4), pro- vidingno evidence of score inflation from related cases."

Discussion¶

Related Claims¶

• Task-specific fine-tuning substantially improves open-source LLMs on MedGame Bench and narrows the gap with commercial models — related
• Without fine-tuning, commercial frontier models outperform open-source LLMs on MedGame Bench structural validity and content metrics — related
• In a pilot with eight medical students, multimodal MedGame was perceived as more engaging and useful than text-only MedGame and original case material — related
• Virtual patient design-variant comparisons rarely show significant differences, with feedback and scaffolding exceptions — a broader claim this one bears on