GPT-4 achieves a pooled MCQ accuracy of 81% on medical licensing examinations, significantly surpassing GPT-3.5's 58%¶

Claim · All claims
Evidence · 1 study · quant-synthesis r3 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q3 i? Across 29 studies, GPT-4's integrated accuracy on medical licensing examination MCQs was 81% (95% CI 78-84), versus 58% (95% CI 53-63) for GPT-3.5 across 25 studies. → Mingxin Liu 2024

Evidence¶

Mingxin Liu 2024¶

Mingxin Liu, Tsuyoshi Okuhara, XinYi Chang, Ritsuko Shirabe, Yuriko Nishiie, Hiroko Okada, Takahiro Kiuchi. (2024). Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis. Journal of Medical Internet Research. https://doi.org/10.2196/60807

q3 · i? · quant-synthesis · r3

Meta-analysis of 25 studies reporting GPT-3.5 accuracy and 29 reporting GPT-4 accuracy, using random-effects models owing to significant heterogeneity (GPT-3.5: I²=95% and GPT-4 I²=93%). The review reports "The integrated accuracy for GPT-3.5 was 58%" and 81% for GPT-4.

"The integrated accuracy for GPT-3.5 was 58% (95% CI 53-63; P<.01), and the integrated accuracy for GPT-4 was 81% (95% CI 78-84; P<.01"

Discussion¶

Related Claims¶

• ChatGPT models achieve a pooled accuracy of 70.1% on national health licensing examinations — related
• Newer OpenAI LLM versions achieve significantly higher pooled accuracy on biomedical MCQs than earlier versions (GPT‐4o–o1: 90%; GPT‐4: 82%; GPT‐3/3.5: 59%) — related
• Subgroup analyses show significant between-model differences, with Copilot (75.25%) and ChatGPT-4 (72.97%) achieving the highest pooled accuracies — related
• LLMs perform worse on dental licensing examinations than on medical licensing examinations — related
• GPT-4 achieves the highest pooled accuracy among LLMs (81%), significantly outperforming GPT-3.5 (60%) — related
• GPT-4 achieves passing accuracy on USMLE Steps 1–3 (mean 86%), significantly exceeding GPT-3.5 (mean 56%) — a narrower finding that bears on this claim
• GPT-3.5 achieves an integrated accuracy of 54% on dental licensing examinations — related
• GPT-4 significantly outperforms ChatGPT-3.5 in accuracy on national health licensing examinations — related
• GPT-4 passed medical licensing examinations in 26 of 29 cases and outperformed average medical students in 13 of 17 cases, while GPT-3.5 passed in only 10 of 26 and surpassed students in 4 of 14 — related