Model and student error patterns on IVDA MCQs show limited question-level consistency but comparable difficulty distributions for easiest and hardest questions¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? The best- and worst-performing questions for models and students overlap minimally, though top questions score above 85% and worst ones below 15% for both. → Markos Stamatakis 2026

Evidence¶

Markos Stamatakis 2026¶

Markos Stamatakis, Omkar Gavali, Joshua Berger, Christian Wartena, Anett Hoppe, and Ralph Ewerth. (2026). Are LLM-based Chatbots Good Enough to Support Computer Science Students in Multiple-Choice Exercises? https://arxiv.org/abs/2606.15919 doi:10.48550/arxiv.2606.15919

q2 · i? · causal · r2

Model–student comparison ranking questions by proportion correct (Table 2, Sec. 4). The article reports "limited consistency between the best- and worst-performing questions" yet comparable difficulty distributions, with top questions above 85% and worst below 15% for both. No effect size printed.

"Table 2 shows limited consistency between the best- and worst-performing questions for models and students, with minimal overlap of questions."

Discussion¶

Related Claims¶

• GPT-4o and GPT-5.2 achieve higher accuracy and more fully correct answers on university IVDA multiple-choice questions than the smaller Mistral-7B and DeepSeek-R1 models — related
• In a two-lecture user study, linear mixed models find no statistically significant effect of GPT-generated output on student MCQ performance, with a tendency toward a group-by-attempt interaction for IVDA — related
• Prompt design significantly influences LLM accuracy on IVDA MCQs, with slide context reducing smaller-model accuracy but improving GPT models, while persona effects are inconsistent — related
• Smaller local LLMs show a minor position bias toward option ordering and partially guessing-like error variation, but accuracy remains relatively stable across permutations and repeated sampling runs — related
• All models predict performance worst for students of intermediate overall ability, with best performance for high and low performing students — related