Blind expert assessment of GPT-4-generated MCQs found the majority valid as exam questions, with problematic-question rates varying by field¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? In Klang et al.'s blind assessment, the majority of GPT-4-generated MCQs were deemed valid exam questions. → Artsi Yaara 2024

Evidence¶

Artsi Yaara 2024¶

Artsi Yaara, Sorin Vera, Konen Eli, Glicksberg Benjamin S., Nadkarni Girish, Klang Eyal. (2024). Large language models for generating medical examinations: systematic review. BMC Medical Education. https://doi.org/10.1186/s12909-024-05239-y

q2 · i? · review · r2

The review reports Klang et al.'s blind assessment of Chat-GPT 4 MCQs across internal medicine, surgery, obstetrics & gynecology, psychiatry, and pediatrics, using few-shot learning from a former exam and five specialist physicians as evaluators who did not know the questions' AI origin.

"The study showed promising results, with the majority of MCQs deemed valid as exam questions"

Discussion¶

Related Claims¶

• All reviewed studies presented faulty LLM-generated questions deemed inappropriate for medical exams — reports the opposite
• Chat-GPT generated more valid physiology MCQs than Bard and Bing, but at a lower difficulty level — related
• Only 40% of ChatGPT-generated dermatology board-style questions were applicable for exams — related
• Most reviewed studies (5 of 8) demonstrated that LLMs can produce MCQs valid for medical exams — a broader claim this one bears on