ChatGPT accuracy on health licensing examinations is highest in pharmacy and lowest in nursing¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q3 i? Pooled accuracy differed significantly by field: pharmacy 71.5%, medicine 69.7%, dentistry 63.2%, nursing 61.8%. → Jin Hye Kyung 2024

Evidence¶

Jin Hye Kyung 2024¶

Jin Hye Kyung, Lee Ha Eun, Kim EunYoung. (2024). Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis. BMC Medical Education. https://doi.org/10.1186/s12909-024-05944-8

q3 · i? · quant-synthesis · r2

Random-effects subgroup analysis by field of health licensing examination. Pooled accuracies were "71.5% (95% CI, 66.3-76.2%) in pharmacy" down to 61.8% in nursing; the difference between fields was significant (Q = 15.334, p = 0.002).

"The effect size for percentage of performance was at an average rate of 71.5% (95% CI, 66.3-76.2%) in pharmacy, 69.7% (95% CI, 65.9-73.2%) in medicine, 63.2% (95% CI, 54.6-71.1%) in dentistry, and 61.8% (95% CI, 58.7-64.9%) in nursing."

Discussion¶

Related Claims¶

• ChatGPT models achieve a pooled accuracy of 70.1% on national health licensing examinations — a broader claim this one bears on
• GPT-4 significantly outperforms ChatGPT-3.5 in accuracy on national health licensing examinations — related
• Removing high-risk-of-bias studies does not materially change the pooled accuracy estimate — related
• Included studies report ChatGPT-3.5 accuracy from 36% to 77% and GPT-4 accuracy from 64.4% to 100% — related
• Large language models achieve a pooled accuracy of 63.69% (95% CI 60.26%-67.06%) in answering dental examination questions — related
• Large language models achieve a pooled accuracy of 72% on medical licensing examination questions across 36 studies — related