In the one human-vs-LLM comparison, human-written MCQs outperformed ChatGPT-generated MCQs in quality and validity, while AI was far faster¶

Claim · All claims
Evidence · 1 study (2 entries) · review r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? Human-written MCQs outperformed Chat-GPT 3.5-generated MCQs, scoring 30 (60%) eligible versus the AI's 20 (40%). → Artsi Yaara 2024 q2 i? Chat-GPT 3.5 generated 50 MCQs in 21 minutes, about 10% of the 211 minutes human writing required. → Artsi Yaara 2024 (2)

Evidence¶

Artsi Yaara 2024¶

Artsi Yaara, Sorin Vera, Konen Eli, Glicksberg Benjamin S., Nadkarni Girish, Klang Eyal. (2024). Large language models for generating medical examinations: systematic review. BMC Medical Education. https://doi.org/10.1186/s12909-024-05239-y

q2 · i? · review · r2

The review reports Cheung et al., the only study comparing LLM to human MCQ writing, evaluated by five international medical experts with no post-generation modification allowed. Humans scored "30 (60%) eligible MCQs" versus the AI's 20 (40%).

"However, the questions written by humans were far better. Both in terms of quality and validity, outperforming the AI in a total score of 30 (60%) eligible MCQs"

Artsi Yaara 2024 (2)¶

Artsi Yaara, Sorin Vera, Konen Eli, Glicksberg Benjamin S., Nadkarni Girish, Klang Eyal. (2024). Large language models for generating medical examinations: systematic review. BMC Medical Education. https://doi.org/10.1186/s12909-024-05239-y

q2 · i? · review · r2

Same Cheung et al. comparison reported in the review: AI generation of 50 MCQs took "21 min", roughly 10% of the 211 minutes required for human writing.

"The overall time required for the AI to generate 50 MCQs was 21 min. This is about 10% of the total time human writing required (211 min)"

Discussion¶

Related Claims¶

• All reviewed studies presented faulty LLM-generated questions deemed inappropriate for medical exams — related
• Chat-GPT generated more valid physiology MCQs than Bard and Bing, but at a lower difficulty level — related
• Faculty-generated SBMCQs met all three ideal item-analysis criteria more often than ChatGPT-generated SBMCQs (19 vs. 9 of 40) — possibly the same claim (merge candidate)
• Human scoring and feedback for HDR takes about 11 minutes 30 seconds per learner, while GPT-based scoring and feedback is nearly instant — related
• GPT-4 achieves the highest pooled accuracy among LLMs (81%), significantly outperforming GPT-3.5 (60%) — related
• Most reviewed studies (5 of 8) demonstrated that LLMs can produce MCQs valid for medical exams — related