Judging learning-objective alignment of generated MCQs is more subjective than answering them, with only moderate human agreement¶

Claim · All claims
Evidence · 2 studies · 2 design r2 · q3
Currency · no dated study recorded · checked against its sources 2026-10-20 (automated, 2 of 2 entries) · not reviewed by a person

Subclaims¶

q3 i? Human judges reached only moderate agreement on whether MCQs test a given learning objective (Fleiss' kappa = 0.58), lower than their agreement on correct answers (0.70). → lo-alignment-agreement-1 q3 i? Gemini aligned slightly better with human judges on LO alignment (mean Cohen's kappa = 0.65) than Phi-2 did (0.62). → lo-alignment-agreement-2

Evidence¶

lo-alignment-agreement-1¶

Yumou Wei, John Stamper, and Paulo F. Carvalho. 2026. Generate-Then-Validate: A Novel Question Generation Approach Using Small Language Models. In LAK26: 16th International Learning Analytics and Knowledge Conference (LAK 2026), Bergen, Norway. ACM, New York, NY, USA. https://doi.org/10.1145/3785022.3785100

q3 · i? · design · r2

In the same 64-MCQ evaluation, judges made binary Yes/No judgments on whether each MCQ tested a specified learning objective under control (original LO) and treatment (different LO) conditions. Agreement was moderate: "Fleiss' kappa = 0.58".

"The seven human judges achieved moderate agreement with each other (Fleiss' kappa = 0.58), indicating that while there was some consensus among the human judges, there were also notable differences in their judgments regarding the alignment of MCQs with learning objectives."

lo-alignment-agreement-2¶

Yumou Wei, John Stamper, and Paulo F. Carvalho. 2026. Generate-Then-Validate: A Novel Question Generation Approach Using Small Language Models. In LAK26: 16th International Learning Analytics and Knowledge Conference (LAK 2026), Bergen, Norway. ACM, New York, NY, USA. https://doi.org/10.1145/3785022.3785100

q3 · i? · design · r2

Average pairwise Cohen's kappa with human judges on LO alignment was 0.65 (SE = 0.02) for Gemini versus 0.62 (SE = 0.08) for Phi-2, against a human-human baseline of 0.58. The article reports agreement statistics, not effect sizes.

"Gemini achieved slightly better alignment with human judges (mean Cohen's kappa = 0.65, SE = 0.02)."

Discussion¶

Related Claims¶

• AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance — the general claim this finding bears on, with the studies behind it
• Raising the answer-confidence threshold reduces the number of retained MCQs but substantially improves their quality — related
• Gemini-2.5-Pro can effectively replicate human expert judgments when answering SLM-generated MCQs — related
• MCQs generated by the generate-then-validate SLM pipeline are generally answerable and unambiguous, as judged by seven human experts — related
• Phi-2's answers to its own generated MCQs align with human experts at least as strongly as the experts agree with each other — related
• ChatGPT answered unsupervised web-based MCQ assessments with a mean accuracy of 70% (range 35–100%) — related
• Student-perceived curricular alignment did not differ between AI and NLE MCQs or among ChatGPT, Gemini and NLE — related
• Inter-annotator agreement was moderate (combined mean Fleiss's κ of 0.62 across 23 groups), yet most students reported having reached consensus — related
• Five judges classifying 63 test units reached average inter-judge correlation of 0.542 ± 0.019 on the two scales — related
• The test's exercises were validated by six expert judges who reached practical agreement after modification — related