ChatGPT-3.5 and ChatGPT-4 both score significantly above the historical MRCP Part 1 pass mark on the sample paper¶
Claim · All claims
Evidence · 1 study (2 entries) · designr2·q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? ChatGPT-3.5 averaged 66.4% on the MRCP UK Part 1 sample paper across four attempts, significantly higher than the historical pass mark (P = 0.0042). → Vij O 2024
q2 i? ChatGPT-4 averaged 84.8% across four attempts, significantly above the historical pass mark (P = 0.0001) and significantly better than ChatGPT-3.5 (P = 0.0001). → Vij O 2024 (2)
Evidence¶
Vij O 2024¶
Vij O, Calver H, Myall N, Dey M, Kouranloo K. (2024). Evaluating the competency of ChatGPT in MRCP Part 1 and a systematic literature review of its capabilities in postgraduate medical assessments. PLOS ONE. https://doi.org/10.1371/journal.pone.0307372
q2 · i? · design · r2
Experimental evaluation in which the 197-question MRCP UK Part 1 sample paper was entered into ChatGPT-3.5 four times and marked against the official mark scheme. The average of 66.4% was "significantly higher than the historical pass mark (P = 0.0042, 95% CI [1.74, 7.05])".
"ChatGPT-3.5 took the MRCP UK Part 1 sample paper four times and scored 126, 136, 128 and 133 out of 197 giving an average percentage of 66.4% (SD 2.32, range 63.4–69.0%) which is significantly higher than the historical pass mark (P = 0.0042, 95% CI [1.74, 7.05])."
Vij O 2024 (2)¶
Vij O, Calver H, Myall N, Dey M, Kouranloo K. (2024). Evaluating the competency of ChatGPT in MRCP Part 1 and a systematic literature review of its capabilities in postgraduate medical assessments. PLOS ONE. https://doi.org/10.1371/journal.pone.0307372
q2 · i? · review · r2
Same four-repeat evaluation with ChatGPT-4, averaging 84.8% out of 197, "significantly above the historical pass mark (P = 0.0001, 95% CI [20.62, 24.97])"; the article also reports ChatGPT-4 significantly better than ChatGPT-3.5 (P = 0.0001, 95% CI [15.25, 21.54]).
"ChatGPT-4 took the MRCP UK Part 1 sample paper four times and scored 164, 168, 169 and 167 out of 197, giving an average percentage of 84.8% (SD 1.10, range 83.2–85.8%) which is significantly above the historical pass mark (P = 0.0001, 95% CI [20.62, 24.97])."
Discussion¶
Related Claims¶
- In the combined review analysis, ChatGPT-3.5 failed eight of nine UK postgraduate exams while ChatGPT-4 passed nine of eleven — reports the opposite
- ChatGPT-4's incorrect MRCP Part 1 answers showed no significant difference between factual recall and complex reasoning question types — related
- MELD analysis found no completion matched the MRCP sample paper at 95% or more, suggesting ChatGPT-4 was unlikely trained on it — related
- ChatGPT-4 outperformed ChatGPT-3.5 in every postgraduate examination both models were tested on — related
- Hallucination, fabrication and reduced explanation accuracy limit ChatGPT's potential as a medical learning tool — related
Page metadata
| Field | Value |
|---|---|
status |
review |
generated |
by process:wiki-ingest on 2026-10-31 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.