Included studies report ChatGPT-3.5 accuracy from 36% to 77% and GPT-4 accuracy from 64.4% to 100%¶

Claim · All claims
Evidence · 1 study · quant-synthesis r3 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across the 23 included studies, reported accuracy ranged from 36 to 77% for ChatGPT-3.5 and 64.4-100% for GPT-4. → Jin Hye Kyung 2024

Evidence¶

Jin Hye Kyung 2024¶

Jin Hye Kyung, Lee Ha Eun, Kim EunYoung. (2024). Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis. BMC Medical Education. https://doi.org/10.1186/s12909-024-05944-8

q2 · i? · quant-synthesis · r3

Descriptive synthesis of the 23 included studies (17 medicine, 3 pharmacy, 2 nursing, 1 dentistry) published 2022-2024 across multiple countries. The review reports "varying accuracy levels, ranging from 36 to 77% for ChatGPT-3.5 and 64.4–100% for GPT-4".

"They reported varying accuracy levels, ranging from 36 to 77% for ChatGPT-3.5 and 64.4–100% for GPT-4."

Discussion¶

Related Claims¶

• GPT-4 achieves the highest pooled accuracy among LLMs (81%), significantly outperforming GPT-3.5 (60%) — a broader claim this one bears on
• Large language models achieve a pooled accuracy of 72% on medical licensing examination questions across 36 studies — a broader claim this one bears on
• ChatGPT answered unsupervised web-based MCQ assessments with a mean accuracy of 70% (range 35–100%) — related
• Subgroup analyses show significant between-model differences, with Copilot (75.25%) and ChatGPT-4 (72.97%) achieving the highest pooled accuracies — related
• ChatGPT accuracy on health licensing examinations is highest in pharmacy and lowest in nursing — related