Most reviewed studies (5 of 8) demonstrated that LLMs can produce MCQs valid for medical exams¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? In a systematic review of eight studies, 5/8 (62.5%) demonstrated valid MCQs generated by LLMs. → Artsi Yaara 2024

Evidence¶

Artsi Yaara 2024¶

Artsi Yaara, Sorin Vera, Konen Eli, Glicksberg Benjamin S., Nadkarni Girish, Klang Eyal. (2024). Large language models for generating medical examinations: systematic review. BMC Medical Education. https://doi.org/10.1186/s12909-024-05239-y

q2 · i? · review · r2

Systematic review of eight studies published April–October 2023, searched in MEDLINE up to November 2023. The review reports that "5/8 (62.5%) studies demonstrated valid MCQs" and that most used Chat-GPT versions 3.5 or 4.

"Overall, 5/8 (62.5%) studies demonstrated valid MCQs. 6/8 (75%) of the studies utilized the latest version Chat-GPT 4"

Discussion¶

Related Claims¶

• Blind expert assessment of GPT-4-generated MCQs found the majority valid as exam questions, with problematic-question rates varying by field — a narrower finding that bears on this claim
• All reviewed studies presented faulty LLM-generated questions deemed inappropriate for medical exams — reports the opposite
• In the one human-vs-LLM comparison, human-written MCQs outperformed ChatGPT-generated MCQs in quality and validity, while AI was far faster — related
• Ontology-based generation produced exam-appropriate case-based MCQs, with 30% of a stratified sample rated appropriate by both expert raters and 50% by at least one — related