Phi-2's answers to its own generated MCQs align with human experts at least as strongly as the experts agree with each other¶

Claim · All claims
Evidence · 2 studies · 2 design r2 · q3
Currency · no dated study recorded · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Pairwise agreement between Phi-2 and human judges on correct answers (mean Cohen's kappa = 0.76) exceeded agreement among human judges themselves (0.70), and Phi-2 matched the human majority with 90.6% accuracy. → phi2-answer-agreement-1

Evidence¶

phi2-answer-agreement-1¶

Yumou Wei, John Stamper, and Paulo F. Carvalho. 2026. Generate-Then-Validate: A Novel Question Generation Approach Using Small Language Models. In LAK26: 16th International Learning Analytics and Knowledge Conference (LAK 2026), Bergen, Norway. ACM, New York, NY, USA. https://doi.org/10.1145/3785022.3785100

q3 · i? · design · r2

In the human evaluation study of 64 generated MCQs, average pairwise Cohen's kappa between Phi-2 and the seven human judges was 0.76 (SE = 0.03) versus a human-human baseline of 0.70. Phi-2's answers were "even more aligned with human experts than the experts were with each other".

"The pairwise agreement between Phi-2 and human judges was higher than that between human judges (mean Cohen's kappa = 0.76, SE = 0.03), suggesting that Phi-2's answers were even more aligned with human experts than the experts were with each other."

phi2-answer-agreement-2¶

Yumou Wei, John Stamper, and Paulo F. Carvalho. 2026. Generate-Then-Validate: A Novel Question Generation Approach Using Small Language Models. In LAK26: 16th International Learning Analytics and Knowledge Conference (LAK 2026), Bergen, Norway. ACM, New York, NY, USA. https://doi.org/10.1145/3785022.3785100

q3 · i? · design · r2

Against the human majority vote treated as ground truth, Phi-2 answered 58 of 64 generated MCQs correctly (90.6% accuracy); both models erred on the same four MCQs. The article reports raw accuracy, not an effect size.

"Phi-2 had an accuracy of 90.6% (58 out of 64), while Gemini had an accuracy of 89.1% (57 out of 64). Both models made the same incorrect predictions on the same four MCQs."

Discussion¶

Related Claims¶

• AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance — the general claim this finding bears on, with the studies behind it
• Raising the answer-confidence threshold reduces the number of retained MCQs but substantially improves their quality — related
• Gemini-2.5-Pro can effectively replicate human expert judgments when answering SLM-generated MCQs — related
• Judging learning-objective alignment of generated MCQs is more subjective than answering them, with only moderate human agreement — related
• MCQs generated by the generate-then-validate SLM pipeline are generally answerable and unambiguous, as judged by seven human experts — a broader claim this one bears on
• ChatGPT answered unsupervised web-based MCQ assessments with a mean accuracy of 70% (range 35–100%) — related
• Human review judged 98.3% of sampled synthetic items aligned with their intended NGSS standards, with no incorrect reference answers — related