No significant difference in item discrimination index or point biserial correlation between ChatGPT 3.5 and human-generated MCQs¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Pairwise meta-analysis found no significant difference in item discrimination index or point biserial correlation between ChatGPT 3.5 and human-generated questions. → Riehm Lauren 2026

Evidence¶

Riehm Lauren 2026¶

Riehm Lauren, Nanji Keean, Lakhani Moiz, Pankiv Evelina, Hasanee Dean, Pfeifer Wesla. (2026). The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis. PLOS One. https://doi.org/10.1371/journal.pone.0340277

q2 · i? · quant-synthesis · r2

Pairwise meta-analysis found "no significant difference in the average item discrimination index" between ChatGPT 3.5 and humans. Point biserial correlation similarly showed no significant difference (MD: −0.05; 95% CI: −0.21 to 0.11).

"Pairwise analysis between ChatGPT 3.5 and humans demonstrated that there is no significant difference in the average item discrimination index (MD −0.07; 95% CI: −0.34, 0.20; GRADE: VERY LOW)"

Discussion¶

Related Claims¶

• ChatPDF.com generates more difficult and discriminating MCQs than ChatGPT 3.5 and ChatGPT 4 — related
• No significant difference in accuracy between ChatGPT 3.5 and Bard 1.0 generated MCQs — related
• Exploratory analysis found item difficulty differed across sources, driven by the Google Gemini versus NLE contrast — related
• A post-search study found no significant difference in item difficulty between AI- and human-generated MCQs, but higher discriminatory power for human questions — related