In a two-lecture user study, linear mixed models find no statistically significant effect of GPT-generated output on student MCQ performance, with a tendency toward a group-by-attempt interaction for IVDA¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Based on likelihood-ratio tests, no fixed effects (attempt, group, group × attempt) are significant at α = 0.05 in either the IVDA or CV scenario. → Markos Stamatakis 2026 q2 i? For IVDA-MCQs the interaction effect approaches significance (LRT p = 0.058), indicating a tendency toward different performance changes between groups. → Markos Stamatakis 2026 (2)

Evidence¶

Markos Stamatakis 2026¶

Markos Stamatakis, Omkar Gavali, Joshua Berger, Christian Wartena, Anett Hoppe, and Ralph Ewerth. (2026). Are LLM-based Chatbots Good Enough to Support Computer Science Students in Multiple-Choice Exercises? https://arxiv.org/abs/2606.15919 doi:10.48550/arxiv.2606.15919

q2 · i? · causal · r2

Linear mixed model analysis (Score ~ Group * Attempt + (1|Participant)) of user-study scores in both scenarios (Table 6, Sec. 5). The article reports "no fixed effects are significant atα= 0.05" under LRT; equivalence was not tested. No effect size printed.

"Based on LRT, no fixed effects are significant atα= 0.05, suggesting no attempt-dependent performance differences between experimental conditions."

Markos Stamatakis 2026 (2)¶

Markos Stamatakis, Omkar Gavali, Joshua Berger, Christian Wartena, Anett Hoppe, and Ralph Ewerth. (2026). Are LLM-based Chatbots Good Enough to Support Computer Science Students in Multiple-Choice Exercises? https://arxiv.org/abs/2606.15919

q2 · i? · causal · r2

LMM interaction test for the IVDA scenario (Table 6, Sec. 5): the group × attempt effect approaches significance with p = 0.058. Under Wald χ2 tests the IVDA interaction (p = 0.046) and CV group effect (p = 0.041) are significant, so evidence is mixed; the article motivates larger studies.

"For IVDA-MCQs, the interaction effect (H3) approaches significance (p= 0.058), indicating a tendency toward different performance changes between the groups, as reflected in Group 3."

Discussion¶

Related Claims¶

• GPT-4o and GPT-5.2 achieve higher accuracy and more fully correct answers on university IVDA multiple-choice questions than the smaller Mistral-7B and DeepSeek-R1 models — related
• Descriptive user-study patterns that the authors read as GPT output possibly scaffolding IVDA students and reducing CV students' performance, an interpretation by prior knowledge that the study did not measure — related
• Model and student error patterns on IVDA MCQs show limited question-level consistency but comparable difficulty distributions for easiest and hardest questions — related
• MCQ accuracy was high (above 87%) across all conditions with no significant difference between diverse and generic explanations — related
• Students perceive LLM chatbots as useful for learning but requiring critical use, reporting hallucinations, overconfidence, and dependency risks — related