Without fine-tuning, commercial frontier models outperform open-source LLMs on MedGame Bench structural validity and content metrics¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Prompted open-source models lag commercial frontier models across structure and content metrics on Medical Narrative Generation and Story Direction. → Qian Wu 2026

Evidence¶

Qian Wu 2026¶

Qian Wu, Xinrong Zhou, Zizhan Ma, Kai Chen, Zheyao Gao, Xun Lin, Hongqiu Wu, Longfei Gou, Yixiao Liu, Ann Sin Nga Lau, Qi Dou. (2026). MedGame: Storytelling Gamification Empowered by Large Language Models for Medical Education. arXiv preprint. https://arxiv.org/abs/2607.21570 doi:10.48550/arxiv.2607.21570

q2 · i? · design · r2

RQ1 benchmark evaluation of prompted baselines (2-shot for narrative generation, zero-shot for story direction). Claude-Sonnet-4.5 and Gemini-3-Pro reached 99.5% and 100.0% Strict validation; open-source Story Direction Sample-wise validation reached only 56.50–80.30% versus over 99% for commercial models.

"Open-source models lag across both structure and content metrics: Qwen3-32B reaches only 79.4% Strict validation, Gemma-3-27B and MedGemma-27B remain below 60%, and their medical accuracy scores are consistently lower than commercial frontier models."

Discussion¶

Related Claims¶

• Task-specific fine-tuning substantially improves open-source LLMs on MedGame Bench and narrows the gap with commercial models — related
• LLM-as-a-Judge scores correlate with blinded human expert judgments, more strongly for Medical Narrative Generation than Story Direction — related
• In a pilot with eight medical students, multimodal MedGame was perceived as more engaging and useful than text-only MedGame and original case material — related
• Related-case overlap between MedGame Bench train and test splits shows no significant performance difference — related
• In one essay-feedback system, fine-tuned models produced no usable deep-level feedback output, so only directly prompted LLMs were evaluated; the authors suggest prompting may be more effective than supervised fine-tuning for this task — related
• Fine-tuned 8B LLMs generate K-2 stories with difficulty-related metric scores statistically significantly different from (mostly better than) zero-shot GPT-4o and Llama 3.3 70B — related