Purpose-built chatbot effects on educational performance are mixed, with some studies showing no benefits or worse outcomes than comparison conditions¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Studies show mixed educational performance outcomes: some improved or comparable learning, others null or inferior effects relative to comparison conditions. → Potel 2026

Evidence¶

Potel 2026¶

Potel, J. T., & Kumashiro, M. (2026). Artificial Intelligence as a Personal Coach: A Narrative Review of Benefits and Risks in Educational and Health Contexts. Behavioral Sciences, 16(8), 1431. https://doi.org/10.3390/bs16081431

q2 · i? · review · r2

Narrative synthesis of educational-performance studies: nursing students with an AI chatbot coach showed no significant differences versus controls; programming students performed best with non-AI digital materials; a story-time chatbot showed lower productivity than controls.

"Thus, these studies do not support a general performance advantage for AI coaching over comparison conditions."

Discussion¶

Related Claims¶

• Purpose-built AI chatbot coaching is generally well received and supports short-term motivation and selected behavior change, but effects on sustained, meaningful outcomes are inconsistent — a broader claim this one bears on
• Purpose-built AI chatbots show generally promising short-term improvements in physical activity and dietary behaviors — related
• One RCT found AI-assisted simulation inferior to standardized patient simulation for breast self-examination skills — related
• Poorly grounded GenAI self-study risks fabricated content and inferior learning relative to curated resources — related
• LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks — a narrower finding that bears on this claim
• Students primarily benefit from AI chatbots in three key areas: homework and study assistance, personalized learning, and skills development — reports the opposite
• AI chatbots in VET show promising effects on self-regulation and task performance — related
• Meta-analysis beyond education found human-AI integration performed worse than the better performer alone in 58% of studies — related
• Evidence for sustained or clinically meaningful outcomes of AI chatbot coaching is mixed and does not guarantee improvement across outcomes — related
• AI-powered simulations show short-term gains in communication confidence and structured knowledge in RCTs, with limited or inferior effects for tactile psychomotor skills — a narrower finding that bears on this claim
• GenAI/LLM modalities provide the strongest evidence for cognitive and communication outcomes, while chatbots show weak independent effects — related