Significant heterogeneity (I2=87.25%) exists among studies of ChatGPT-4.0 accuracy in medical licensing exams¶

Claim · All claims
Evidence · 1 study · quant-synthesis r3 · q3
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q3 i? There was significant heterogeneity among studies, with I2=87.25% and χ216=125.54 (P<.001), suggesting differences in study design, population, and exam content influenced accuracy estimates. → Lesselroth Blake 2025

Evidence¶

Lesselroth Blake 2025¶

Lesselroth Blake, Riese Janik, Yang Zhichao, Jaleel Shahila, Shabbir Waqas, Jaleel Anila, Aziz Umair, Farid Ghulam, Zahid Bashir Muhammad, Mirza Tehmasp Rehman, Khizar Abbas Syed Mohammad, Aslam Shiraz, Sikander Rana Muhammad Hassaan. (2025). Evaluating the Potential and Accuracy of ChatGPT-3.5 and 4.0 in Medical Licensing and In-Training Examinations: Systematic Review and Meta-Analysis. JMIR Medical Education. https://doi.org/10.2196/68070

q3 · i? · quant-synthesis · r3

The meta-analysis found "significant heterogeneity among the studies, with I2=87.25% and χ216=125.54 (P<.001)", indicating high variability across studies of ChatGPT-4.0 licensing exam accuracy.

"However, there was significant heterogeneity among the studies, with I2=87.25% and χ216=125.54 (P<.001)."

Discussion¶

Related Claims¶

• ChatGPT-4.0 was 36% more likely than ChatGPT-3.5 to provide correct answers across medical licensing and residency exams — related
• ChatGPT-4.0 outperforms ChatGPT-3.5 in medical licensing exams, achieving a pooled accuracy of 81.8% versus 60.8% — related