ChatGPT-4.0 was 36% more likely than ChatGPT-3.5 to provide correct answers across medical licensing and residency exams¶
Claim · All claims
Evidence · 1 study · quant-synthesisr2·q3
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? The forest plot showed a risk ratio of 1.36 (95% CI 1.30-1.43), indicating ChatGPT-4 was 36% more likely to provide correct answers than ChatGPT-3.5. → Lesselroth Blake 2025
Evidence¶
Lesselroth Blake 2025¶
Lesselroth Blake, Riese Janik, Yang Zhichao, Jaleel Shahila, Shabbir Waqas, Jaleel Anila, Aziz Umair, Farid Ghulam, Zahid Bashir Muhammad, Mirza Tehmasp Rehman, Khizar Abbas Syed Mohammad, Aslam Shiraz, Sikander Rana Muhammad Hassaan. (2025). Evaluating the Potential and Accuracy of ChatGPT-3.5 and 4.0 in Medical Licensing and In-Training Examinations: Systematic Review and Meta-Analysis. JMIR Medical Education. https://doi.org/10.2196/68070
q3 · i? · quant-synthesis · r2
Meta-analysis across both exam types found "a risk ratio of 1.36 (95% CI 1.30-1.43)" favoring ChatGPT-4 over ChatGPT-3.5. The article prints a risk ratio rather than a standardized effect size, so impact is null.
"The forest plot presented a risk ratio of 1.36 (95% CI 1.30-1.43), demonstrating that ChatGPT-4 was 36% more likely to provide correct answers than ChatGPT-3.5 across both medical licensing and residency exams."
Discussion¶
Related Claims¶
- In in-training residency exams, ChatGPT-4.0 achieved 72.2% accuracy compared to 57.7% for ChatGPT-3.5 — related
- Large language models achieve a pooled accuracy of 72% on medical licensing examination questions across 36 studies — a broader claim this one bears on
- Subgroup analyses show significant between-model differences, with Copilot (75.25%) and ChatGPT-4 (72.97%) achieving the highest pooled accuracies — related
- No evidence of publication bias is found in the meta-analysis of LLM examination accuracy — related
- Significant heterogeneity (I2=87.25%) exists among studies of ChatGPT-4.0 accuracy in medical licensing exams — related
Page metadata
| Field | Value |
|---|---|
status |
review |
generated |
by process:wiki-ingest on 2026-10-31 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.