ChatGPT-4's incorrect MRCP Part 1 answers showed no significant difference between factual recall and complex reasoning question types¶
Claim · All claims
Evidence · 1 study · reviewr2·q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Of the 16 questions ChatGPT-4 got incorrect across four attempts, 4 were factual recall and 12 clinical reasoning, not statistically significant relative to their proportions in the sample paper (Z = 1.0263, P = 0.30302). → Vij O 2024
Evidence¶
Vij O 2024¶
Vij O, Calver H, Myall N, Dey M, Kouranloo K. (2024). Evaluating the competency of ChatGPT in MRCP Part 1 and a systematic literature review of its capabilities in postgraduate medical assessments. PLOS ONE. https://doi.org/10.1371/journal.pone.0307372
q2 · i? · review · r2
Two-proportion Z test on ChatGPT-4's 16 incorrect MRCP Part 1 questions (4 factual recall, 12 clinical reasoning) against the question-type proportions in the sample paper; no statistically significant difference, and equivalence was not tested.
"This was not found to be statistically significant compared with the proportion of factual recall and clinical reasoning questions in the MRCP Part 1 sample questions (Z = 1.0263, P = 0.30302)."
Discussion¶
Related Claims¶
- ChatGPT-3.5 and ChatGPT-4 both score significantly above the historical MRCP Part 1 pass mark on the sample paper — related
- LLMs perform worse on complex, open-ended, or subjective assignments than on short, clearly formulated answers — a broader claim this one bears on
- General-purpose LLMs show accuracy and reliability limits, including hallucinations and task-specific accuracy below perfect, making them risky as expert advice sources — related
- ChatGPT-4 outperformed ChatGPT-3.5 in every postgraduate examination both models were tested on — related
- Task nature modulates brain activation: problem-solving tasks and difficult cases induced higher activation than simple recall or routine cases, and incorrect answers induced higher PFC activation than correct answers — related
Page metadata
| Field | Value |
|---|---|
status |
review |
generated |
by process:wiki-ingest on 2026-10-31 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.