Chat-GPT generated more valid physiology MCQs than Bard and Bing, but at a lower difficulty level¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? In a cross-sectional comparison of Chat-GPT 3.5, Bard, and Bing, Chat-GPT's MCQs were significantly more valid, but less difficult than Bard's and Bing's. → Artsi Yaara 2024

Evidence¶

Artsi Yaara 2024¶

Artsi Yaara, Sorin Vera, Konen Eli, Glicksberg Benjamin S., Nadkarni Girish, Klang Eyal. (2024). Large language models for generating medical examinations: systematic review. BMC Medical Education. https://doi.org/10.1186/s12909-024-05239-y

q2 · i? · design · r2

The review reports Agarwal et al.'s cross-sectional study comparing Chat-GPT 3.5, Bard, and Bing on reasoning-based physiology MCQs referenced to the Indian NMC 11-module curriculum. The review reports validity and difficulty differences between the named tools without reproducing the printed p values.

"Chat-GPT's generated MCQs were significantly more valid than the other AI tools examined in the study. However, the difficulty level was lower compared to Bard and Bing"

Discussion¶

Related Claims¶

• No significant difference in accuracy between ChatGPT 3.5 and Bard 1.0 generated MCQs — reports the opposite
• In the one human-vs-LLM comparison, human-written MCQs outperformed ChatGPT-generated MCQs in quality and validity, while AI was far faster — related
• Blind expert assessment of GPT-4-generated MCQs found the majority valid as exam questions, with problematic-question rates varying by field — related