ChatGPT-4 outperformed ChatGPT-3.5 in every postgraduate examination both models were tested on¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? ChatGPT-4 outperformed ChatGPT-3.5 in all examinations taken by both models, with the difference reported as statistically significant in four examinations including MRCP Part 1 (P<0.001). → Vij O 2024

Evidence¶

Vij O 2024¶

Vij O, Calver H, Myall N, Dey M, Kouranloo K. (2024). Evaluating the competency of ChatGPT in MRCP Part 1 and a systematic literature review of its capabilities in postgraduate medical assessments. PLOS ONE. https://doi.org/10.1371/journal.pone.0307372

q2 · i? · review · r2

Combined analysis drawing on the included studies and this study's MRCP result; the article reports the ChatGPT-4 advantage was "significantly greater in four examinations, including MRCP Part 1 (P<0.001)".

"ChatGPT-4 outperformed ChatGPT-3.5 in all examinations that were taken by both LLMs. This was reported to be significantly greater in four examinations, including MRCP Part 1 (P<0.001) [37,42,43]."

Discussion¶

Related Claims¶

• In the combined review analysis, ChatGPT-3.5 failed eight of nine UK postgraduate exams while ChatGPT-4 passed nine of eleven — related
• GPT-4 achieves the highest pooled accuracy among LLMs (81%), significantly outperforming GPT-3.5 (60%) — a broader claim this one bears on
• GPT-4 significantly outperforms ChatGPT-3.5 in accuracy on national health licensing examinations — a narrower finding that bears on this claim
• ChatGPT-3.5 and ChatGPT-4 both score significantly above the historical MRCP Part 1 pass mark on the sample paper — related
• ChatGPT-4's incorrect MRCP Part 1 answers showed no significant difference between factual recall and complex reasoning question types — related