Ambient consumer AI use in the holdout did not close the structured-tutor gap¶
Claim · All claims
Evidence · 1 study · causalr2·q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? Holdout students used consumer AI freely (21–42% in any given week), yet the tutor's randomized advantage held, and removing ambient-AI users from the control group left the estimate essentially unchanged. → Shihao Yang 2026
Evidence¶
Shihao Yang 2026¶
Shihao Yang, Marshall Van Alstyne, and Chrysanthos Dellarocas. (2026). When AI Tutors Speak: Evidence from a Randomized Field Experiment. https://arxiv.org/abs/2609.23958 doi:10.48550/arxiv.2609.23958
q3 · i? · causal · r2
Weekly holdout check-in surveys measured consumer-AI tool use (ChatGPT, Gemini, Claude, and similar) as a manipulation check and preregistered contamination covariate, defining the estimand as the tutor's marginal value over a realistic self-study environment.
"Holdout students used consumer AI freely throughout the study (21–42% in any given week), yet the tutor's random-ized advantage holds over that realistic environment, and removing ambient-AI users from the control group leaves the estimate essentially unchanged."
Discussion¶
Related Claims¶
- Access to a structured conversational AI tutor increased post-test content scores over a no-AI holdout with ambient consumer AI available — related
- The instructor's own final exam showed the same direction of advantage on the intact randomized sample, with no difference on the pre-treatment midterm — related
- A quarter of students offered a free AI tutor never tried it once, making first-session adoption a deployment problem — related
- The tutor-access learning gain was concentrated in conceptual depth of written answers rather than multiple-choice scores — related
- Reviews citing one randomised trial (Bastani et al. 2025) report that unrestricted GPT-4 access in high-school mathematics raised assisted practice performance but left students worse off after withdrawal — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
ambient-consumer-ai-did-not-close-tutor-gap |
title |
Ambient consumer AI use in the holdout did not close the structured-tutor gap |
description |
Ambient consumer AI use in the holdout did not close the structured-tutor gap |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — shihao-yang-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.