Design AI coaching systems to preserve user agency, recognize vulnerability, and stop sycophantic encouragement during maladaptive thought processes¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The review recommends safeguards including that AI systems respond appropriately to signs of vulnerability and direct users to professional support, and that AI "should be programmed to recognize and stop sycophantic encouragement when users engage in maladaptive and distressing thought processes". It further recommends public education about hallucinations, sycophantic responses, and dependence risk, and design that preserves agency while providing warm, friendly, accurate advice. AI coaching should be clearly distinguished from counseling and clinical care.

Design Implications¶

Context¶

Requirements¶

• Transparency about system limitations per WHO guidance; human oversight; recognition of when users should be directed to professional support

Constraints¶

• Earlier recommendations to increase emotional engagement by anthropomorphizing AI may now increase risk of unhealthy attachment with general-purpose LLMs

Target Learners¶

• users of AI coaching including those experiencing loneliness or distress
• general public using LLMs for advice

Target Learning Goals¶

• safe personal development support and minimizing harm from inaccurate or excessively affirming AI guidance

Affordances¶

• Chatbot Technology Working Classification

Related Strategies¶

• Use inclusive design, calibrated supports, and clear delegation norms to preserve learner agency and epistemic diversity
• Design AI systems as relational translators and adopt friction-preserving designs to buffer asymmetric risks
• Calibrate assistant support forms to preserve users' cognitive participation rather than making every interaction tutor-like

Examples¶

-

Key Sources¶

• Potel, J. T., & Kumashiro, M. (2026). Artificial Intelligence as a Personal Coach: A Narrative Review of Benefits and Risks in Educational and Health Contexts. Behavioral Sciences, 16(8), 1431. https://doi.org/10.3390/bs16081431