Access to a structured conversational AI tutor increased post-test content scores over a no-AI holdout with ambient consumer AI available¶
Claim · All claims
Evidence · 1 study · causalr2·q3
Currency · newest study 2026 · checked against its sources 2026-10-19 (automated, 1 of 1 entry) · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? Students randomized to the structured AI tutor scored higher on the 55-point post-test than ability-matched holdout peers, an intent-to-treat effect of about 6.6 points (p=.007). → Shihao Yang 2026
Evidence¶
Shihao Yang 2026¶
Shihao Yang, Marshall Van Alstyne, and Chrysanthos Dellarocas. (2026). When AI Tutors Speak: Evidence from a Randomized Field Experiment. https://arxiv.org/abs/2609.23958 doi:10.48550/arxiv.2609.23958
q3 · i? · causal · r2
Complete-case randomized comparison in the preregistered field experiment (86 randomized, 51 post-test completers; analysis sample N=46). The article reports the primary ANCOVA estimate as +6.63 (p=.007) and this inverse-probability-weighted re-estimate, "gives+6.52(p=.005)".
"re-estimating the primary specification with inverse-probability weights (inclusion modeled on arm and pre-score) gives+6.52(p=.005)"
Discussion¶
Related Claims¶
- AI-supported instruction has a moderate positive effect on student achievement in STEM education (g = 0.670 under a random effects model) — the general claim this finding bears on, with the studies behind it
- Ambient consumer AI use in the holdout did not close the structured-tutor gap — related
- A quarter of students offered a free AI tutor never tried it once, making first-session adoption a deployment problem — related
- The instructor's own final exam showed the same direction of advantage on the intact randomized sample, with no difference on the pre-treatment midterm — related
- The tutor-access learning gain was concentrated in conceptual depth of written answers rather than multiple-choice scores — a narrower finding that bears on this claim
- Multi-agent AI standardized patient training improved final OSCE-aligned examination scores compared with a structured non-LLM control condition — related
- Students learn more with a research-based AI tutor than with in-class active learning on identical content (post-test scores) — related
- AI access raises immediate unaided test scores by 7.2 percentage points (0.28 SD) after a timed learning phase — related
- In math problem solving, a tutor-plus-peers LLM arrangement produced the strongest unassisted post-task test accuracy, with accuracy increasing monotonically from Control to Tutor + Peers — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
structured-ai-tutor-raises-post-test-scores-field-experiment |
title |
Access to a structured conversational AI tutor increased post-test content scores over a no-AI holdout with ambient consumer AI available |
description |
Access to a structured conversational AI tutor increased post-test content scores over a no-AI holdout with ambient consumer AI available |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
source_checked |
2026-10-19 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — shihao-yang-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.