GPT-4o and GPT-5.2 achieve higher accuracy and more fully correct answers on university IVDA multiple-choice questions than the smaller Mistral-7B and DeepSeek-R1 models¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across 12 prompt variants on 70 IVDA MCQs, GPT models reach higher accuracy and fully-correct rates (GPT-5.2 average 91.2% accuracy, 71.3% fully correct) than Mistral-7B (69.2%, 26.0%) and DeepSeek-R1 (75.2%, 38.1%). → Markos Stamatakis 2026

Evidence¶

Markos Stamatakis 2026¶

Markos Stamatakis, Omkar Gavali, Joshua Berger, Christian Wartena, Anett Hoppe, and Ralph Ewerth. (2026). Are LLM-based Chatbots Good Enough to Support Computer Science Students in Multiple-Choice Exercises? https://arxiv.org/abs/2606.15919 doi:10.48550/arxiv.2606.15919

q2 · i? · causal · r2

Benchmark evaluation of four chatbots on 70 IVDA MCQs across 12 prompt variants (Table 1). The article reports that "the GPT models achieve higher accuracy and more fully correct responses", with GPT-5.2 outperforming GPT-4o; smaller models select incorrect options for most questions. No effect size is printed.

"In contrast, the GPT models achieve higher accuracy and more fully correct responses, likely due to their larger scale and training."

Discussion¶

Related Claims¶

• Prompt design significantly influences LLM accuracy on IVDA MCQs, with slide context reducing smaller-model accuracy but improving GPT models, while persona effects are inconsistent — related
• Smaller local LLMs show a minor position bias toward option ordering and partially guessing-like error variation, but accuracy remains relatively stable across permutations and repeated sampling runs — related
• Descriptive user-study patterns that the authors read as GPT output possibly scaffolding IVDA students and reducing CV students' performance, an interpretation by prior knowledge that the study did not measure — related
• In a two-lecture user study, linear mixed models find no statistically significant effect of GPT-generated output on student MCQ performance, with a tendency toward a group-by-attempt interaction for IVDA — related
• Model and student error patterns on IVDA MCQs show limited question-level consistency but comparable difficulty distributions for easiest and hardest questions — related
• LLM-simulated student ideas moderately overlap with teacher- and student-generated lesson ideas (mean cosine similarity 0.52), with larger LLMs slightly outperforming Mistral-7B — related
• LLM accuracy increases monotonically with prompt sophistication across semantically equivalent medical questions, with Low Literacy users scoring significantly below Prompt Engineer users — related
• Newer LLM generations such as GPT-4 generally outperform GPT-3.5 in short-answer grading accuracy, but this advantage is not consistent across all tasks and domains — related
• The author reports that prompt phrasing and the specific foundational model had secondary impact on final material quality compared with the structure of the workflow itself. — related
• ChatGPT answered unsupervised web-based MCQ assessments with a mean accuracy of 70% (range 35–100%) — related
• Larger Llama Guard models outperform smaller ones on education-prompt classification, with the 8B model best in accuracy, recall, and F1 — related
• Model and prompt choice account for only a small share of misalignment error, which concentrates in transcript-conditioned higher-order interactions — related