Only 40% of ChatGPT-generated dermatology board-style questions were applicable for exams¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? In Ayub et al.'s dermatology study, only 16 (40%) of generated questions were applicable; the rest were unclear, inaccurate, or of low complexity. → Artsi Yaara 2024

Evidence¶

Artsi Yaara 2024¶

Artsi Yaara, Sorin Vera, Konen Eli, Glicksberg Benjamin S., Nadkarni Girish, Klang Eyal. (2024). Large language models for generating medical examinations: systematic review. BMC Medical Education. https://doi.org/10.1186/s12909-024-05239-y

q2 · i? · review · r2

The review reports Ayub et al., which used Chat-PDF to upload JAAD CME articles into Chat-GPT 3.5 and had two board-certified dermatologists evaluate 40 questions on accuracy, complexity, and clarity. "Only 16 (40%)" were applicable.

"The three parameters to evaluate the MCQs were accuracy, complexity, and clarity. Only 16 (40%) of the generated questions were applicable"

Discussion¶

Related Claims¶

• General-purpose LLMs show accuracy and reliability limits, including hallucinations and task-specific accuracy below perfect, making them risky as expert advice sources — a broader claim this one bears on
• Generative AI chatbots answer orthopaedic exam questions with moderate accuracy, below trainee performance — related
• LLMs perform worse on complex, open-ended, or subjective assignments than on short, clearly formulated answers — a broader claim this one bears on
• Blind expert assessment of GPT-4-generated MCQs found the majority valid as exam questions, with problematic-question rates varying by field — related