Faculty-generated SBMCQs met all three ideal item-analysis criteria more often than ChatGPT-generated SBMCQs (19 vs. 9 of 40)¶

Claim · All claims
Evidence · 1 study · causal r2 · q3
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Among 40 items per source, 19 faculty SBMCQs (47.5%) versus 9 ChatGPT SBMCQs (22.5%) met ideal criteria of appropriate difficulty, high discrimination, and 100% distractor effectiveness. → Chauhan 2025

Evidence¶

Chauhan 2025¶

Chauhan, A., Khaliq, F., & Nayak, K. R. (2025). Assessing Quality of Scenario-Based Multiple-Choice Questions in Physiology: Faculty-Generated vs. ChatGPT-Generated Questions among Phase I Medical Students. International Journal of Artificial Intelligence in Education, 35, 2315–2344. https://doi.org/10.1007/s40593-025-00471-z

q3 · i? · causal · r2

Item analysis of four test sessions in a crossover study of 120 first-year MBBS students, each answering 20 faculty and 20 ChatGPT items. The article reports "19 (47.5%) of the SBMCQs met all three criteria" for faculty versus 9 (22.5%) for ChatGPT; no effect size or significance test is printed.

"In our study,19 (47.5%) of the SBMCQs met all three criteria for an ideal MCQ: appropriate Difficulty Index (30–70%), high Discrimination Index (≥ 0.25), and 100% Distractor Effectiveness. In contrast, the number of ChatGPT-generated SBMCQs meeting these criteria was 9(22.5%)."

Discussion¶

Related Claims¶

• ChatGPT-generated SBMCQs contained a higher rate of nonfunctional distractors than faculty-generated SBMCQs (33.75% vs. 13.75%) — related
• Faculty focus groups raised concerns about authenticity, validity, and reliability of ChatGPT-generated SBMCQs, including cultural and linguistic bias — related
• AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance — possibly the same claim (merge candidate)
• Faculty-generated SBMCQs showed superior discrimination indices and distractor effectiveness compared with ChatGPT-generated SBMCQs — related
• ChatGPT-generated SBMCQs skewed easier than faculty-generated SBMCQs, whose difficulty was predominantly in the appropriate range — related