Prompt design significantly influences LLM accuracy on IVDA MCQs, with slide context reducing smaller-model accuracy but improving GPT models, while persona effects are inconsistent¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Providing slide-based context constantly reduces Mistral-7B accuracy to 62–69% while approaches without content reach over 70%, whereas GPT models improve when slide content is provided. → Markos Stamatakis 2026 q2 i? Different prompting approaches significantly influence results, but no consistent effect is observed for personas. → Markos Stamatakis 2026 (2)

Evidence¶

Markos Stamatakis 2026¶

Markos Stamatakis, Omkar Gavali, Joshua Berger, Christian Wartena, Anett Hoppe, and Ralph Ewerth. (2026). Are LLM-based Chatbots Good Enough to Support Computer Science Students in Multiple-Choice Exercises? https://arxiv.org/abs/2606.15919 doi:10.48550/arxiv.2606.15919

q2 · i? · causal · r2

Benchmark results for Mistral-7B across prompt variants (Table 1, Sec. 4). The article reports that "Providing slide con- tent constantly reduces the accuracy to62−69 %", interpreting larger input as hindering identification of question-relevant information. Descriptive percentages only; no effect size printed.

"Providing slide con- tent constantly reduces the accuracy to62−69 %. Approaches without con- tent reach over70 %, suggesting larger input hinders identification of question- relevant information."

Markos Stamatakis 2026 (2)¶

Markos Stamatakis, Omkar Gavali, Joshua Berger, Christian Wartena, Anett Hoppe, and Ralph Ewerth. (2026). Are LLM-based Chatbots Good Enough to Support Computer Science Students in Multiple-Choice Exercises? https://arxiv.org/abs/2606.15919

q2 · i? · causal · r2

Authors' summary of the 12-prompt benchmark in Sec. 4: prompting approaches significantly influence results while persona effects show no consistent pattern. The article prints no effect size for this statement.

"Furthermore, the different prompting approaches significantly influence the results, while no consistent effect is observed for personas (RQ2)."

Discussion¶

Related Claims¶

• GPT-4o and GPT-5.2 achieve higher accuracy and more fully correct answers on university IVDA multiple-choice questions than the smaller Mistral-7B and DeepSeek-R1 models — related
• Curricular chain-of-thought prompting improves competency-classification accuracy over zero-shot, with gains concentrated in larger models, while definition-based prompting does not improve performance — related
• Prompt quality and detailed scoring rubrics significantly influence the accuracy and consistency of LLM-generated grades — related
• Model and student error patterns on IVDA MCQs show limited question-level consistency but comparable difficulty distributions for easiest and hardest questions — related
• LLM accuracy increases monotonically with prompt sophistication across semantically equivalent medical questions, with Low Literacy users scoring significantly below Prompt Engineer users — related