Hallucination, fabrication and reduced explanation accuracy limit ChatGPT's potential as a medical learning tool¶

Claim · All claims
Evidence · 1 study (2 entries) · review r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? The review reports, citing Ghosn et al., that GPT-4's explanation accuracy in the FRCR Part 2A examination was 65.9%, below its correct-answer rate of 80.5%. → Vij O 2024 q1 i? The review reports, citing Walters and Wilder, that 18% of GPT-4 citations in a generated short literature review were fabricated. → Vij O 2024 (2)

Evidence¶

Vij O 2024¶

Vij O, Calver H, Myall N, Dey M, Kouranloo K. (2024). Evaluating the competency of ChatGPT in MRCP Part 1 and a systematic literature review of its capabilities in postgraduate medical assessments. PLOS ONE. https://doi.org/10.1371/journal.pone.0307372

q2 · i? · review · r2

The review attributes this to Ghosn et al.'s FRCR Part 2A comparison: explanation accuracy of 65.9% versus a correct-answer rate of 80.5%, suggesting ChatGPT-4 may not explain answers correctly at a passing level.

"This propensity for misinformation is further exemplified by LLM’s poor explanation accuracy; GPT-4 demonstrated an explanation accuracy of 65.9% dropping from the correct rate of 80.5% in the FRCR Part 2A examination [42]."

Vij O 2024 (2)¶

Vij O, Calver H, Myall N, Dey M, Kouranloo K. (2024). Evaluating the competency of ChatGPT in MRCP Part 1 and a systematic literature review of its capabilities in postgraduate medical assessments. PLOS ONE. https://doi.org/10.1371/journal.pone.0307372

q2 · i? · review · r2

The review reports, citing prior studies, that ChatGPT inaccurately reports publication content and that "18% of GPT-4 citations were fabricated" in a generated short literature review, illustrating fabrication risk for learners.

"ChatGPT has been shown to inaccurately report the content of genuine publications and when asked to generate a short literature review, 18% of GPT-4 citations were fabricated [52,53]."

Discussion¶

Related Claims¶

• ChatGPT-3.5 and ChatGPT-4 both score significantly above the historical MRCP Part 1 pass mark on the sample paper — related
• Poorly grounded GenAI self-study risks fabricated content and inferior learning relative to curated resources — a narrower finding that bears on this claim
• AI-assisted scientific writing assignments risk fabricated references and altered chemical meaning — a narrower finding that bears on this claim
• Students perceive LLM chatbots as useful for learning but requiring critical use, reporting hallucinations, overconfidence, and dependency risks — related
• In the combined review analysis, ChatGPT-3.5 failed eight of nine UK postgraduate exams while ChatGPT-4 passed nine of eleven — related
• General-purpose LLMs show accuracy and reliability limits, including hallucinations and task-specific accuracy below perfect, making them risky as expert advice sources — a broader claim this one bears on
• The most frequently cited ethical concerns about GenAI in research were hallucinated facts, fabricated references, and misleading content — related
• ChatGPT fabricates references: the authors caught it inventing a non-existent 'Ribeiro and Vala, 2020' article with full bibliographic details — a narrower finding that bears on this claim
• Students broadly endorse LLMs as educational aids (74% positive) while 61% raise integrity concerns, 59% cite hallucination risk, and 82% report no institutional guidance — related