Large language models achieve a pooled accuracy of 72% on medical licensing examination questions across 36 studies¶

Claim · All claims
Evidence · 1 study · quant-synthesis r3 · q3
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q3 i? Across 36 included studies evaluating 63 LLMs, pooled accuracy on medical licensing examination questions was 72% (95% CI, 70.0%–75.0%). → Nouri Haniyeh 2025

Evidence¶

Nouri Haniyeh 2025¶

Nouri Haniyeh, Mahdavi Abdollah, Abedi Ali, Mohammadnia Alireza, Hamedan Mahnaz, Amanzadeh Masoud. (2025). Performance of large language models in medical licensing examinations: a systematic review and meta-analysis. Journal of Educational Evaluation for Health Professions. https://doi.org/10.3352/jeehp.2025.22.36

q3 · i? · quant-synthesis · r3

Random-effects meta-analysis of 36 studies (2,404 records screened) evaluating LLM accuracy on medical licensing examination questions. The review reports "The pooled accuracy of all models was 72%" with very high heterogeneity (I2=99.99%).

"The pooled accuracy of all models was 72% (95% CI, 70.0%–75.0%). The heterogeneity across studies was significant (I2=99.99%, P<0.001)."

Discussion¶

Related Claims¶

• GPT-4 achieves the highest pooled accuracy among LLMs (81%), significantly outperforming GPT-3.5 (60%) — related
• Pooled LLM accuracy differs minimally by question source (71% for internal exams vs. 73% for public databases) — related
• Studies using both text- and image-based questions show higher pooled LLM accuracy (78%) than text-only studies (71%) — related
• No evidence of publication bias is found in the meta-analysis of LLM examination accuracy — related
• LLM accuracy does not differ significantly across question languages, ranging from 62% in Polish to 77% in German — related
• Subgroup analyses show significant between-model differences, with Copilot (75.25%) and ChatGPT-4 (72.97%) achieving the highest pooled accuracies — a narrower finding that bears on this claim
• ChatGPT-4.0 outperforms ChatGPT-3.5 in medical licensing exams, achieving a pooled accuracy of 81.8% versus 60.8% — a narrower finding that bears on this claim
• In in-training residency exams, ChatGPT-4.0 achieved 72.2% accuracy compared to 57.7% for ChatGPT-3.5 — related
• ChatGPT-4.0 was 36% more likely than ChatGPT-3.5 to provide correct answers across medical licensing and residency exams — a narrower finding that bears on this claim
• GPT-4 outperforms GPT-3.5 and Bard on dental licensing examinations, achieving an integrated accuracy of 72% — a narrower finding that bears on this claim
• LLMs perform worse on dental licensing examinations than on medical licensing examinations — related