ChatGPT models achieve a pooled accuracy of 70.1% on national health licensing examinations¶
Claim · All claims
Evidence · 1 study · quant-synthesisr2·q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? Across 23 studies of national licensing examinations, the overall pooled accuracy of ChatGPT models was 70.1% (95% CI, 65.0-74.8%). → Jin Hye Kyung 2024
Evidence¶
Jin Hye Kyung 2024¶
Jin Hye Kyung, Lee Ha Eun, Kim EunYoung. (2024). Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis. BMC Medical Education. https://doi.org/10.1186/s12909-024-05944-8
q3 · i? · quant-synthesis · r2
Random-effects meta-analysis of 23 studies (144 effect sizes) of ChatGPT-3.5 and GPT-4 accuracy on national licensing examinations in medicine, pharmacy, dentistry, and nursing. The pooled estimate was "70.1% (95% CI, 65.0-74.8%)"; heterogeneity was high (Q = 1201.303, I2 = 98.2%).
"The overall effect size for the percentage of performance of ChatGPT models was 70.1% (95% CI, 65.0-74.8%)."
Discussion¶
Related Claims¶
- ChatGPT accuracy on health licensing examinations is highest in pharmacy and lowest in nursing — a narrower finding that bears on this claim
- GPT-4 significantly outperforms ChatGPT-3.5 in accuracy on national health licensing examinations — related
- GPT-4 outperforms GPT-3.5 and Bard on dental licensing examinations, achieving an integrated accuracy of 72% — related
- Large language models achieve a pooled accuracy of 72% on medical licensing examination questions across 36 studies — a broader claim this one bears on
- GPT-4 achieves the highest pooled accuracy among LLMs (81%), significantly outperforming GPT-3.5 (60%) — related
- Large language models achieve a pooled accuracy of 63.69% (95% CI 60.26%-67.06%) in answering dental examination questions — related
- Subgroup analyses show significant between-model differences, with Copilot (75.25%) and ChatGPT-4 (72.97%) achieving the highest pooled accuracies — related
- GPT-4 achieves a pooled MCQ accuracy of 81% on medical licensing examinations, significantly surpassing GPT-3.5's 58% — related
Page metadata
| Field | Value |
|---|---|
status |
review |
generated |
by process:wiki-ingest on 2026-10-31 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.