GPT-4 performed significantly better on examinations from English-speaking countries (86% vs 80%, P=.02), while GPT-3.5 showed no such difference (P=.72)¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q3 i? In subgroup 1 meta-analysis, GPT-4's pooled accuracy was 86% (95% CI 82-89) for English-speaking countries versus 80% (95% CI 76-83) for non-English-speaking countries in original languages, a statistically significant difference (P=.02); GPT-3.5 showed 57% versus 58% with no significant difference (P=.72). → Mingxin Liu 2024

Evidence¶

Mingxin Liu 2024¶

Mingxin Liu, Tsuyoshi Okuhara, XinYi Chang, Ritsuko Shirabe, Yuriko Nishiie, Hiroko Okada, Takahiro Kiuchi. (2024). Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis. Journal of Medical Internet Research. https://doi.org/10.2196/60807

q3 · i? · quant-synthesis · r2

Subgroup 1 meta-analysis with random-effects models due to heterogeneity (GPT-4: I²=69% and 93%). The review reports "Statistically significant differences were observed between the results (P=.02)" for GPT-4, while GPT-3.5's 57% versus 58% difference was not significant (P=.72).

"For GPT-4, the integrated accuracy in examinations from English-speaking countries was 86% (95% CI 82%-89%; P<.01), and in examinations from non–English-speaking countries with original languages, it was 80% (95% CI 76-83; P<.01). Statistically significant differences were observed between the results (P=.02"

Discussion¶

Related Claims¶

• GPT-3.5 performs significantly better on dental licensing examinations from English-speaking countries than non-English-speaking countries — reports the opposite
• GPT-4's performance on dental licensing examinations does not differ significantly between English-speaking and non-English-speaking countries — reports the opposite
• LLMs perform worse on dental licensing examinations than on medical licensing examinations — related
• The review concludes GPT-4 is not yet suitable for use in medical education despite its potential, due to insufficient accuracy, inconsistent performance, and country-specific policy and knowledge challenges — a broader claim this one bears on
• GPT-4 significantly outperforms ChatGPT-3.5 in accuracy on national health licensing examinations — related