In the combined review analysis, ChatGPT-3.5 failed eight of nine UK postgraduate exams while ChatGPT-4 passed nine of eleven¶

Claim · All claims
Evidence · 1 study (2 entries) · quant-synthesis r1 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? ChatGPT-3.5 failed eight out of nine postgraduate UK medical exams, averaging 5.0% below the pass mark. → Vij O 2024 q2 i? ChatGPT-4 passed nine out of eleven examinations, averaging 13.56% above the pass mark, failing only FRCS (Orth) Part 1 and FRCR Part 1. → Vij O 2024 (2)

Evidence¶

Vij O 2024¶

Vij O, Calver H, Myall N, Dey M, Kouranloo K. (2024). Evaluating the competency of ChatGPT in MRCP Part 1 and a systematic literature review of its capabilities in postgraduate medical assessments. PLOS ONE. https://doi.org/10.1371/journal.pone.0307372

q2 · i? · quant-synthesis · r1

Combined analysis of the 12 included SLR studies plus this study's MRCP result, summarising ChatGPT-3.5's performance across nine postgraduate UK exams with an average score of 56% and an average difference from the pass mark of -5.0%.

"ChatGPT-3.5 failed eight out of nine postgraduate UK medical exams it performed, and only achieved a mark within or above passing range in MRCP Part 1. The average score for ChatGPT-3.5 was 56% (SD 0.084, range 43%-69.7%), with an average difference between GPT-3.5 score and pass mark of -5.0%."

Vij O 2024 (2)¶

Vij O, Calver H, Myall N, Dey M, Kouranloo K. (2024). Evaluating the competency of ChatGPT in MRCP Part 1 and a systematic literature review of its capabilities in postgraduate medical assessments. PLOS ONE. https://doi.org/10.1371/journal.pone.0307372

q2 · i? · quant-synthesis · r1

Combined analysis across eleven examinations (Table 1) showing ChatGPT-4 passed nine, averaging 76.2% with a +13.56% difference from the pass mark; the failures were FRCS (Orth) Part 1 and FRCR Part 1, which require 6 and 2 years of postgraduate experience respectively.

"ChatGPT-4 passed nine out of eleven examinations it performed with an average score of 76.2% (SD 1.23, range 50%-100%), with an average difference between GPT-4 score and pass mark of +13.56%. The specialties that ChatGPT-4 failed were orthopaedics and radiology."

Discussion¶

Related Claims¶

• ChatGPT-4 outperformed ChatGPT-3.5 in every postgraduate examination both models were tested on — related
• ChatGPT-3.5 and ChatGPT-4 both score significantly above the historical MRCP Part 1 pass mark on the sample paper — reports the opposite
• GPT-4 exceeds pass marks on India NEET-PG and UK PLAB 1 but falls below pass on clinical reasoning and procedural items — related
• GPT-4 achieves passing accuracy on USMLE Steps 1–3 (mean 86%), significantly exceeding GPT-3.5 (mean 56%) — related
• GPT-4 examination accuracy is moderated by item type: multi-step vignettes (61–71%) and image-dependent items (48–63%) fall well below single-fact recall (83–88%) — related
• GPT-4 passed medical licensing examinations in 26 of 29 cases and outperformed average medical students in 13 of 17 cases, while GPT-3.5 passed in only 10 of 26 and surpassed students in 4 of 14 — related
• Hallucination, fabrication and reduced explanation accuracy limit ChatGPT's potential as a medical learning tool — related