MCQs generated by the generate-then-validate SLM pipeline are generally answerable and unambiguous, as judged by seven human experts¶

Claim · All claims
Evidence · 1 study · design r2 · q3
Currency · no dated study recorded · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Seven middle-school science teachers showed substantial agreement on the correct answers to 64 generated MCQs, and only 1.6% of the MCQs were deemed unanswerable by majority vote. → human-answers-1

Evidence¶

human-answers-1¶

Yumou Wei, John Stamper, and Paulo F. Carvalho. 2026. Generate-Then-Validate: A Novel Question Generation Approach Using Small Language Models. In LAK26: 16th International Learning Analytics and Knowledge Conference (LAK 2026), Bergen, Norway. ACM, New York, NY, USA. https://doi.org/10.1145/3785022.3785100

q3 · i? · design · r2

Human evaluation study: seven middle-school science teachers answered 64 MCQs generated from eight AP Environmental Science learning objectives, each with a 'none of the above' option added. Their Fleiss' kappa of 0.70 indicates "the generated MCQs are generally answerable and unambiguous".

"The seven human judges achieved a substantial agreement [21] with each other (Fleiss' kappa = 0.70), indicating that the generated MCQs are generally answerable and unambiguous."

Discussion¶

Related Claims¶

• AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance — the general claim this finding bears on, with the studies behind it

• Gemini-2.5-Pro can effectively replicate human expert judgments when answering SLM-generated MCQs — related

• Judging learning-objective alignment of generated MCQs is more subjective than answering them, with only moderate human agreement — related
• Phi-2's answers to its own generated MCQs align with human experts at least as strongly as the experts agree with each other — a narrower finding that bears on this claim
• Raising the answer-confidence threshold reduces the number of retained MCQs but substantially improves their quality — related
• Human review judged 98.3% of sampled synthetic items aligned with their intended NGSS standards, with no incorrect reference answers — related
• Expert review establishes excellent content validity for TAILS (S-CVI/Ave = 0.98; I-CVI 0.86–1.00) — related
• Model size defines a core trade-off: too-large SLMs answer without the cognitive structure while too-small SLMs break instruction-following coherence