AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance¶

Claim · All claims
Evidence · 1 study (2 entries) · review r2–r3 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Only 9 of 40 ChatGPT physiology items met all ideal criteria compared with 19 of 40 faculty items. → Zhao J 2026 q2 i? AI item generation reduced authoring time and sometimes produced difficulty and discrimination comparable with human items, but incorrect keys, nonfunctional distractors, and poor discrimination required faculty correction. → Zhao J 2026 (2)

Evidence¶

Zhao J 2026¶

Zhao J, Yang K, Gao H and Jiang G (2026). Generative artificial intelligence and the transformation of medical education: a scoping review. Front. Educ. 11:1907589. https://doi.org/10.3389/feduc.2026.1907589

q2 · i? · review · r3

The review reports this psychometric comparison from Chauhan et al. within the 22-report assessment category as evidence of inconsistent AI item performance.

"For example, only 9 of 40 ChatGPT physiology items met all ideal criteria compared with 19 of 40 faculty items (Chauhan et al., 2025)."

Zhao J 2026 (2)¶

Zhao J, Yang K, Gao H and Jiang G (2026). Generative artificial intelligence and the transformation of medical education: a scoping review. Front. Educ. 11:1907589. https://doi.org/10.3389/feduc.2026.1907589

q2 · i? · review · r2

The review reports these positive signals alongside settings where incorrect keys, multiple defensible answers, nonfunctional distractors, and mismatched cognitive levels required faculty correction.

"AI item generation reduced authoring time and sometimes produced difficulty and discrimination comparable with human items (Al-Najafi et al., 2026; Wu et al., 2026)."

Studies bearing on this claim¶

Can a language model write usable multiple-choice items and distractors, or do its drafts need human review? This table gathers the studies behind this claim and the 17 narrower claims that bear on it: 8 distinct studies (3 design, 2 associational, 1 review, 1 qualitative, 1 causal), q2–q3, 0 of 8 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• Students' main AI challenges are content reliability and depth, and most expect AI to serve as tutor, creative partner, and critical thinking partner rather than mere content generator — related
• Educators use AI tools mainly for personalized learning and language support and frequently use generative AI for lesson planning — related
• 75% of student-authored physics questions met combined high-quality criteria (clear, correct, plausible distractors, above recall, adequate explanation) — related
• Benchmark items showed high mean difficulty and low item discrimination, with many trivial items and only one broken item — related
• AI pilots found models not yet capable of high-quality math output generation, including multiple choice problems, coherent word problems, and math visuals — related
• PeerWise student ratings of multiple-choice questions correlate well with instructor ratings (0.5 and 0.58) — related
• Judging learning-objective alignment of generated MCQs is more subjective than answering them, with only moderate human agreement — related