Gemini-2.5-Pro can effectively replicate human expert judgments when answering SLM-generated MCQs¶

Claim · All claims
Evidence · 1 study · associational r? · q3
Currency · no dated study recorded · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Gemini achieved the same mean pairwise agreement with human judges on correct answers as Phi-2 (Cohen's kappa = 0.76) and 89.1% accuracy against the human majority, supporting its use as a surrogate judge. → gemini-answer-agreement-1

Evidence¶

gemini-answer-agreement-1¶

Yumou Wei, John Stamper, and Paulo F. Carvalho. 2026. Generate-Then-Validate: A Novel Question Generation Approach Using Small Language Models. In LAK26: 16th International Learning Analytics and Knowledge Conference (LAK 2026), Bergen, Norway. ACM, New York, NY, USA. https://doi.org/10.1145/3785022.3785100

q3 · i? · associational · r?

In the automatic evaluation study, Gemini-2.5-Pro evaluated the same 64 MCQs under the same within-subject design as the human study, via two-turn conversations with probability-extracting scaffolds. The authors conclude Gemini "can effectively replicate human judgments".

"The high agreement between human judges and Gemini also indicated that Gemini can effectively replicate human judgments in answering MCQs generated from our pipeline, highlighting its potential as a reliable surrogate judge for the expansive generation stage of our pipeline."

Discussion¶

Related Claims¶

• Raising the answer-confidence threshold reduces the number of retained MCQs but substantially improves their quality — related
• Judging learning-objective alignment of generated MCQs is more subjective than answering them, with only moderate human agreement — related
• Phi-2's answers to its own generated MCQs align with human experts at least as strongly as the experts agree with each other — related
• MCQs generated by the generate-then-validate SLM pipeline are generally answerable and unambiguous, as judged by seven human experts — related
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• A Gemini-based creativity autorater scores real students' complex multimedia creativity tasks on par with human experts (item Kappa 0.66; total-score Pearson r = 0.88) — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related
• An LLM-based analogy judge validated against expert judgments shows moderate-to-strong agreement and screens most generated analogies as meeting baseline adequacy — related
• In the blind judicial-exam evaluation, Gemini 2.5 Pro scored 21/24, above the top human benchmark of 18/24, while the other LLMs scored substantially lower. — related
• Humans are more certain when judging that facilitation is not needed than when judging it is — related
• The LLM-judge scoring pipeline for the two metrics aligns with human labels and human raters — related