ChatPDF.com generates more difficult and discriminating MCQs than ChatGPT 3.5 and ChatGPT 4¶
Claim · All claims
Evidence · 1 study · quant-synthesisr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? ChatPDF.com had a lower difficulty index (easier questions) and higher item discrimination index than ChatGPT 3.5 and ChatGPT 4. → Riehm Lauren 2026
Evidence¶
Riehm Lauren 2026¶
Riehm Lauren, Nanji Keean, Lakhani Moiz, Pankiv Evelina, Hasanee Dean, Pfeifer Wesla. (2026). The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis. PLOS One. https://doi.org/10.1371/journal.pone.0340277
q2 · i? · quant-synthesis · r2
Subgroup analysis found significant differences in difficulty index (chi-squared: 116.72; df = 3, p < 0.01). Post-hoc tests showed "ChatPDF.com had a lower difficulty index than ChatGPT 3.5 (p < 0.001), ChatGPT 4 (p = 0.010), and humans (p < 0.001)". ChatPDF.com also had higher discrimination.
"Exploratory post-hoc t-tests demonstrated that ChatPDF.com had a lower difficulty index than ChatGPT 3.5 (p < 0.001), ChatGPT 4 (p = 0.010), and humans (p < 0.001)."
Discussion¶
Related Claims¶
- No significant difference in item discrimination index or point biserial correlation between ChatGPT 3.5 and human-generated MCQs — related
- Faculty-generated SBMCQs met all three ideal item-analysis criteria more often than ChatGPT-generated SBMCQs (19 vs. 9 of 40) — related
- ChatGPT-generated SBMCQs skewed easier than faculty-generated SBMCQs, whose difficulty was predominantly in the appropriate range — related
- AI-generated exam questions were somewhat easier than standardized-test questions, with average difficulty βAI = −0.45 versus βSTD = 0.35 — related
- Faculty-generated SBMCQs showed superior discrimination indices and distractor effectiveness compared with ChatGPT-generated SBMCQs — related
- AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance — related
- AI-generated questions showed slightly higher average discrimination than standardized test questions (αAI = 1.3 vs αSTD = 1.2), with moderate evidence — related
Page metadata
| Field | Value |
|---|---|
status |
review |
generated |
by process:wiki-ingest on 2026-10-31 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.