GPT-4 significantly outperforms ChatGPT-3.5 in accuracy on national health licensing examinations¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q3 i? In subgroup analysis, GPT-4 achieved significantly higher accuracy (80.4%) than ChatGPT-3.5 (58.9%) on national licensing examinations. → Jin Hye Kyung 2024

Evidence¶

Jin Hye Kyung 2024¶

Jin Hye Kyung, Lee Ha Eun, Kim EunYoung. (2024). Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis. BMC Medical Education. https://doi.org/10.1186/s12909-024-05944-8

q3 · i? · quant-synthesis · r2

Random-effects subgroup analysis comparing the two ChatGPT model versions across the included studies. GPT-4 pooled accuracy was "80.4% (95% CI, 78.6-82.0%)" versus 58.9% for ChatGPT-3.5; the difference was significant (Q = 177.027, p < 0.001).

"The effect size for the percentage of performance with regard to the ChatGPT models was 58.9% (95% CI, 56.2-61.6%) for ChatGPT-3.5 and 80.4% (95% CI, 78.6-82.0%) for GPT-4."

Discussion¶

Related Claims¶

• ChatGPT-4 outperformed ChatGPT-3.5 in every postgraduate examination both models were tested on — a broader claim this one bears on
• ChatGPT accuracy on health licensing examinations is highest in pharmacy and lowest in nursing — related
• ChatGPT models achieve a pooled accuracy of 70.1% on national health licensing examinations — related
• GPT-4 achieves a pooled MCQ accuracy of 81% on medical licensing examinations, significantly surpassing GPT-3.5's 58% — related
• GPT-4 achieves the highest pooled accuracy among LLMs (81%), significantly outperforming GPT-3.5 (60%) — possibly the same claim (merge candidate)
• ChatGPT-4.0 outperforms ChatGPT-3.5 in medical licensing exams, achieving a pooled accuracy of 81.8% versus 60.8% — possibly the same claim (merge candidate)
• Large language models achieve a pooled accuracy of 72% on medical licensing examination questions across 36 studies — a broader claim this one bears on
• Subgroup analyses show significant between-model differences, with Copilot (75.25%) and ChatGPT-4 (72.97%) achieving the highest pooled accuracies — related
• LLMs perform worse on dental licensing examinations than on medical licensing examinations — related
• GPT-4 performed significantly better on examinations from English-speaking countries (86% vs 80%, P=.02), while GPT-3.5 showed no such difference (P=.72) — related