ChatGPT-4.0 outperforms ChatGPT-3.5 in medical licensing exams, achieving a pooled accuracy of 81.8% versus 60.8%¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q3
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q3 i? ChatGPT-4.0 achieved a pooled accuracy of 81.8% in medical licensing exams, compared to ChatGPT-3.5's 60.8%. → Lesselroth Blake 2025

Evidence¶

Lesselroth Blake 2025¶

Lesselroth Blake, Riese Janik, Yang Zhichao, Jaleel Shahila, Shabbir Waqas, Jaleel Anila, Aziz Umair, Farid Ghulam, Zahid Bashir Muhammad, Mirza Tehmasp Rehman, Khizar Abbas Syed Mohammad, Aslam Shiraz, Sikander Rana Muhammad Hassaan. (2025). Evaluating the Potential and Accuracy of ChatGPT-3.5 and 4.0 in Medical Licensing and In-Training Examinations: Systematic Review and Meta-Analysis. JMIR Medical Education. https://doi.org/10.2196/68070

q3 · i? · quant-synthesis · r2

Meta-analysis of 53 studies using a random-effects inverse-variance model found ChatGPT-4.0 "achieving a pooled accuracy of 81.8%, compared to ChatGPT-3.5's 60.8%" in medical licensing exams. No standardized effect size is printed for this contrast, so impact is null.

"ChatGPT-4 outperformed ChatGPT-3.5 in medical licensing exams, achieving a pooled accuracy of 81.8%, compared to ChatGPT-3.5's 60.8%."

Discussion¶

Related Claims¶

• GPT-4 achieves the highest pooled accuracy among LLMs (81%), significantly outperforming GPT-3.5 (60%) — possibly the same claim (merge candidate)
• Large language models achieve a pooled accuracy of 72% on medical licensing examination questions across 36 studies — a broader claim this one bears on
• Subgroup analyses show significant between-model differences, with Copilot (75.25%) and ChatGPT-4 (72.97%) achieving the highest pooled accuracies — related
• In in-training residency exams, ChatGPT-4.0 achieved 72.2% accuracy compared to 57.7% for ChatGPT-3.5 — a narrower finding that bears on this claim
• Pooled LLM accuracy differs minimally by question source (71% for internal exams vs. 73% for public databases) — related
• GPT-4 achieves passing accuracy on USMLE Steps 1–3 (mean 86%), significantly exceeding GPT-3.5 (mean 56%) — a narrower finding that bears on this claim
• ChatGPT performed at or near the passing threshold on the United States Medical Licensing Exam without domain fine-tuning (reported from Kung et al.) — related
• Significant heterogeneity (I2=87.25%) exists among studies of ChatGPT-4.0 accuracy in medical licensing exams — related
• LLMs perform worse on dental licensing examinations than on medical licensing examinations — related