A student-facing chatbot was pulled after it reinforced a student's self-deprecating views about math, illustrating substantial risk of direct AI engagement with students¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? One cohort team removed its student-facing chatbot prototype after learning it reinforced a student's self-deprecating views about not being good at math. → chatbot

Evidence¶

chatbot¶

Noakes, S., Shell, A., Murillo, A., Van Nostrand, P., Ruiz, P., Cornick, S., and Karim, S. (2024, October). An Ethical and Equitable Vision of AI in Education: Learning Across 28 Exploratory Projects. Digital Promise. https://doi.org/10.51388/20.500.12265/232

q2 · i? · design · r2

Technical finding on chatbots with students, reported in Table 5. The report notes success was found in restricting a bot to asking probing questions to gather information for one output, in this case a word problem, to minimize toxicity.

"There is still substantial risk to engaging directly with students. One team developing a student-facing bot pulled the prototype as soon as they were alerted to the bot reinforcing a student's self-deprecating views on not being good at math."

Discussion¶

Related Claims¶

• Dialogic competence is a prerequisite for meaningful engagement with LLM-based chatbots — related
• Using AI to evaluate AI outputs was often challenging because AI act as yes people and humans struggled to agree on evaluation rubrics — related
• Instructors authoring AI tutoring chatbots in a MOOC almost never systematically tested their bots before deploying them to students — related
• Students perceive LLM chatbots as useful for learning but requiring critical use, reporting hallucinations, overconfidence, and dependency risks — related