Distractor distribution did not differ across ChatGPT, Google Gemini and NLE MCQs¶

Claim · All claims
Evidence · 1 study · associational r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? A three-way comparison of distractor distribution showed no significant differences across MCQ origins. → Vogt P 2026

Evidence¶

Vogt P 2026¶

Vogt P, Wolf N, Jordan S, Volz-Willems S, Jäger J, Dupont F. (2026). Can students identify AI? – A cross-sectional quantitative study about AI recognition in tablet-based MCQ assessment among fifth-year undergraduate medical students at Saarland University, Germany. BMC Medical Education. https://doi.org/10.1186/s12909-026-10410-8

q3 · i? · associational · r2

Exploratory Kruskal–Wallis test of the distractor distribution index across ChatGPT (Mdn = 0.840), Google Gemini (Mdn = 0.725) and NLE (Mdn = 0.820) items. The three-way comparison showed "no significant differences" and the combined AI vs NLE comparison also did not differ (U = 394, p = 0.529, r = 0.095).

"No significant differences in a three-way compari - son occurred (χ 2 (2) = 2.61, p = 0.271; ε2 = 0.045). Com- bined AI comparison with NLE distractor distribution did not differ (AI: Mdn = 0.760, IQR 0.390; n = 29 vs."

Discussion¶

Related Claims¶

• AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance — the general claim this finding bears on, with the studies behind it
• Perceived overall utility of generative AI differs significantly across the five platforms used, favoring platforms with structured task breakdowns — related
• Student-perceived curricular alignment did not differ between AI and NLE MCQs or among ChatGPT, Gemini and NLE — related
• Exploratory analysis found item difficulty differed across sources, driven by the Google Gemini versus NLE contrast — related