Expert-in-the-loop revision substantially improves medical accuracy and educational quality of generated storylines¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Full expert revision of low-scoring generated story drafts raises Medical Accuracy and Educational quality scores for both evaluated models. → Qian Wu 2026

Evidence¶

Qian Wu 2026¶

Qian Wu, Xinrong Zhou, Zizhan Ma, Kai Chen, Zheyao Gao, Xun Lin, Hongqiu Wu, Longfei Gou, Yixiao Liu, Ann Sin Nga Lau, Qi Dou. (2026). MedGame: Storytelling Gamification Empowered by Large Language Models for Medical Education. arXiv preprint. https://arxiv.org/abs/2607.21570 doi:10.48550/arxiv.2607.21570

q2 · i? · design · r2

Expert-in-the-loop refinement study on 10 low-scoring generated story drafts per model from Gemini-3-Pro and Qwen3.5-27B∗; experts marked targeted revision regions, averaging 8.3 and 8.9 regions per sample. No effect size printed.

"As shown in Table 16, full expert revision substan- tially improves both medical accuracy and educa- tional quality: for Gemini, the average Medical Accuracy / Educational scores increase from 6.40 / 6.45 to 8.07 / 7.95; for Qwen3.5-27B∗, they in- crease from 6.20 / 6.35 to 7.87 / 7.75."

Discussion¶

Related Claims¶

• LLM-as-a-Judge scores correlate with blinded human expert judgments, more strongly for Medical Narrative Generation than Story Direction — related
• LLM-based rubric scores agreed with expert ratings at the total-score level and on cognitive-process dimensions, but agreement was low for more subjective dimensions — related
• Prompt quality and detailed scoring rubrics significantly influence the accuracy and consistency of LLM-generated grades — related