Raising the answer-confidence threshold reduces the number of retained MCQs but substantially improves their quality¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · no dated study recorded · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? In the ablation study, raising the answer-confidence threshold from baseline to 95% halved retained MCQs (6,043 to about 2,856) while Phi-2–Gemini agreement on correct answers rose from kappa 0.67 to 0.91. → ablation-threshold-1

Evidence¶

ablation-threshold-1¶

Yumou Wei, John Stamper, and Paulo F. Carvalho. 2026. Generate-Then-Validate: A Novel Question Generation Approach Using Small Language Models. In LAK26: 16th International Learning Analytics and Knowledge Conference (LAK 2026), Bergen, Norway. ACM, New York, NY, USA. https://doi.org/10.1145/3785022.3785100

q2 · i? · causal · r2

Ablation study across all 41 learning objectives, repeated ten times with different random seeds, using Gemini as surrogate judge. Retained MCQs fell from 6,043 at baseline to about 2,856 at the 95% threshold while agreement improved from "0.67 at the baseline to 0.91 at the 95% threshold".

"The average Cohen's kappa between Phi-2 and Gemini improved from 0.67 at the baseline to 0.91 at the 95% threshold, indicating a substantial increase in agreement on the correct answers. This suggests that the answer confidence evaluation step was effective."

Discussion¶

Related Claims¶

• AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance — the general claim this finding bears on, with the studies behind it
• Judging learning-objective alignment of generated MCQs is more subjective than answering them, with only moderate human agreement — related
• Phi-2's answers to its own generated MCQs align with human experts at least as strongly as the experts agree with each other — related
• Gemini-2.5-Pro can effectively replicate human expert judgments when answering SLM-generated MCQs — related
• MCQs generated by the generate-then-validate SLM pipeline are generally answerable and unambiguous, as judged by seven human experts — related
• Higher seed-problem filtering thresholds yield better downstream reasoning performance, indicating problem quality directly impacts results — related