Instructors authoring AI tutoring chatbots in a MOOC almost never systematically tested their bots before deploying them to students¶
Claim · All claims
Evidence · 1 study · associationalr1·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? In a formative study of 121 chatbots created by instructors in an AI for Educators MOOC, nearly all failed to engage in any systematic testing before publication. → Miina Koyama 2026
Evidence¶
Miina Koyama 2026¶
Miina Koyama, Ruiwei Xiao, and John Stamper. (2026). PromptDecipher: Supporting AI Tutor Authoring Through Editable Simulated Interactions. https://arxiv.org/abs/2605.16605 doi:10.48550/arxiv.2605.16605
q2 · i? · associational · r1
Formative observational study of 121 chatbots created by instructors in an "AI for Educators" MOOC. The article reports that "while teachers successfully specified learning content, nearly all failed to engage in any systematic testing before publication"; no effect size is printed.
"Our formative study of 121 chatbots created by instructors in an "AI for Educators" MOOC corroborates this at scale: while teachers successfully specified learning content, nearly all failed to engage in any systematic testing before publication"
Discussion¶
Related Claims¶
- A student-facing chatbot was pulled after it reinforced a student's self-deprecating views about math, illustrating substantial risk of direct AI engagement with students — related
- Log-based evaluation showed chatbot alignment was stronger for responsiveness (88.9%) and persona (81.5%) than for rules (70.4%) and purpose (59.3%) — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
teachers-rarely-test-ai-tutor-bots-before-publication |
title |
Instructors authoring AI tutoring chatbots in a MOOC almost never systematically tested their bots before deploying them to students |
description |
Instructors authoring AI tutoring chatbots in a MOOC almost never systematically tested their bots before deploying them to students |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — miina-koyama-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.