A question-repetition bug in Quiz Mode, reported by 2/8 pilot participants, was a defect simulation-based evaluation could not identify¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? A question-repetition bug was reported by 2/8 participants, one classifying it as severe and reporting it prevented meaningful use. → Rumble 2026

Evidence¶

Rumble 2026¶

Rumble, T., Zarrin, J., Lovell, P. G., Falconer, R. (2026). Learning Engagement Assistant (LEA): Cross-Course Scalability and Classroom Evaluation of an Agentic AI Tutoring System. arXiv. https://arxiv.org/abs/2607.13370 doi:10.48550/arxiv.2607.13370

q2 · i? · design · r2

Pilot survey technical-reliability findings from the eight-participant CMP511 study. The authors state the defect is one that "simulation-based evaluation is structurally unable to identify", making it a deployment-only failure mode.

"However, a question-repetition bug was reported by 2/8 participants; one classified it as "severe" and reported that it prevented meaningful use. This defect is one that simulation-based evaluation is structurally unable to identify."

Discussion¶

Related Claims¶

• Quiz mode's course-blind Python question generation was the only cross-course failure requiring a code change during onboarding — related
• In an exploratory pilot (n=8), students rated LEA's usability, Chat Mode, Quiz Mode, learning outcomes, and trust mostly positively, with Chat Mode the strongest performer — related
• In simulation, LEA met or exceeded targets in Chat and Tutor modes while Quiz Mode's KC Alignment Score fell below target — related
• Tutor Mode's adaptive responsiveness was overestimated by simulation: only 3/8 pilot participants agreed LEA adapted appropriately when they expressed uncertainty — related