LLM-as-a-Judge scores correlate with blinded human expert judgments, more strongly for Medical Narrative Generation than Story Direction¶

Claim · All claims
Evidence · 1 study · design r2 · q2 · i3 large
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i3 Average correlation between LLM judge and human ratings was r = 0.81 for Medical Narrative Generation and r = 0.61 for Story Direction. → Qian Wu 2026

Evidence¶

Qian Wu 2026¶

Qian Wu, Xinrong Zhou, Zizhan Ma, Kai Chen, Zheyao Gao, Xun Lin, Hongqiu Wu, Longfei Gou, Yixiao Liu, Ann Sin Nga Lau, Qi Dou. (2026). MedGame: Storytelling Gamification Empowered by Large Language Models for Medical Education. arXiv preprint. https://arxiv.org/abs/2607.21570 doi:10.48550/arxiv.2607.21570

q2 · i3 · design · r2

Human validation of the LLM-as-a-Judge protocol: three medical Ph.D. holders rated 50 Medical Narrative Generation cases and two game developers rated 50 Story Direction cases using the same rubrics as GPT-5.2, blinded to model identities. Correlations r = 0.81 and r = 0.61.

"LLM scores correlate with human judgments, with strong agreement for Medical Narrative Generation (avg.r= 0.81 ) and moderate agreement for Story Direction (avg.r= 0.61)."

Discussion¶

Related Claims¶

• Without fine-tuning, commercial frontier models outperform open-source LLMs on MedGame Bench structural validity and content metrics — related
• Expert-in-the-loop revision substantially improves medical accuracy and educational quality of generated storylines — related
• Human raters' preferences align with the LLM judge on TutorBench (Pearson r = 0.82 across metric-level win rates) — a narrower finding that bears on this claim
• A Gemini-based creativity autorater scores real students' complex multimedia creativity tasks on par with human experts (item Kappa 0.66; total-score Pearson r = 0.88) — related
• The cross-family LLM judge panel agrees with human domain experts on every judged axis, reaching the expert ceiling on answer-holding — related
• LLM-based rubric scores agreed with expert ratings at the total-score level and on cognitive-process dimensions, but agreement was low for more subjective dimensions — related
• In a pilot with eight medical students, multimodal MedGame was perceived as more engaging and useful than text-only MedGame and original case material — related
• CLARA annotations align with blinded educator judgments, with 0.81 pairwise agreement and highest-rated educational interpretability — related
• CLARA achieves the highest agreement with blinded human developmental ranking judgments compared with readability and prompting baselines — related
• Task-specific fine-tuning substantially improves open-source LLMs on MedGame Bench and narrows the gap with commercial models — related
• An LLM-based analogy judge validated against expert judgments shows moderate-to-strong agreement and screens most generated analogies as meeting baseline adequacy — related