GPT-4 passed medical licensing examinations in 26 of 29 cases and outperformed average medical students in 13 of 17 cases, while GPT-3.5 passed in only 10 of 26 and surpassed students in 4 of 14¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? GPT-4 passed in 89.7% (26/29) of studies and surpassed medical students in 76.5% (13/17); GPT-3.5 passed or performed satisfactorily in 38.5% (10/26) and surpassed students in 28.6% (4/14). → Mingxin Liu 2024

Evidence¶

Mingxin Liu 2024¶

Mingxin Liu, Tsuyoshi Okuhara, XinYi Chang, Ritsuko Shirabe, Yuriko Nishiie, Hiroko Okada, Takahiro Kiuchi. (2024). Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis. Journal of Medical Internet Research. https://doi.org/10.2196/60807

q2 · i? · quant-synthesis · r2

Narrative synthesis of the 45 included studies: for GPT-3.5, 6 passed and 4 performed satisfactorily (38.5%, 10/26); for GPT-4, "17 reported that GPT-4 passed the medical licensing examination, and 9 reported satisfactory performance" (89.7%). Student comparisons: GPT-4 surpassed students in 13/17 (76.5%), GPT-3.5 in 4/14 (28.6%).

"Among the 29 studies testing GPT-4, 17 reported that GPT-4 passed the medical licensing examination, and 9 reported satisfactory performance, making up 89.7% (26/29) of the total."

Discussion¶

Related Claims¶

• GPT-4 achieves a pooled MCQ accuracy of 81% on medical licensing examinations, significantly surpassing GPT-3.5's 58% — related
• GPT-4 achieves passing accuracy on USMLE Steps 1–3 (mean 86%), significantly exceeding GPT-3.5 (mean 56%) — related
• ChatGPT's accuracy on image-based MCQs ranged from 13.1% to 100% across studies, and it performed significantly worse on open-ended questions than on MCQs — related
• In the combined review analysis, ChatGPT-3.5 failed eight of nine UK postgraduate exams while ChatGPT-4 passed nine of eleven — related
• ChatGPT-4.0 outperforms ChatGPT-3.5 in medical licensing exams, achieving a pooled accuracy of 81.8% versus 60.8% — related