Faculty-generated SBMCQs showed superior discrimination indices and distractor effectiveness compared with ChatGPT-generated SBMCQs¶

Claim · All claims
Evidence · 1 study · causal r2 · q3
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Faculty items better distinguished high from low performers: faculty discrimination indices were predominantly 'good' [16] and 'excellent' [14], while ChatGPT items included 9 'poor' and 8 'marginal'. → Chauhan 2025

Evidence¶

Chauhan 2025¶

Chauhan, A., Khaliq, F., & Nayak, K. R. (2025). Assessing Quality of Scenario-Based Multiple-Choice Questions in Physiology: Faculty-Generated vs. ChatGPT-Generated Questions among Phase I Medical Students. International Journal of Artificial Intelligence in Education, 35, 2315–2344. https://doi.org/10.1007/s40593-025-00471-z

q3 · i? · causal · r2

Discussion statement summarizing the discrimination-index classification across the four test sessions (Table 4), where faculty items were predominantly 'good' [16] and 'excellent' [14] versus 9 'poor' and 8 'marginal' ChatGPT items; Table 5 reports mean DI 0.33 vs. 0.18. No significance test is printed.

"Faculty-generated SBMCQs demonstrated superior discrimination indices and distractor effectiveness compared to those generated by ChatGPT (Table 4)."

Discussion¶

Related Claims¶

• ChatGPT-generated SBMCQs contained a higher rate of nonfunctional distractors than faculty-generated SBMCQs (33.75% vs. 13.75%) — related
• Faculty focus groups raised concerns about authenticity, validity, and reliability of ChatGPT-generated SBMCQs, including cultural and linguistic bias — related
• Faculty-generated SBMCQs met all three ideal item-analysis criteria more often than ChatGPT-generated SBMCQs (19 vs. 9 of 40) — related
• ChatGPT-generated SBMCQs skewed easier than faculty-generated SBMCQs, whose difficulty was predominantly in the appropriate range — related
• AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance — related