The instructor's own final exam showed the same direction of advantage on the intact randomized sample, with no difference on the pre-treatment midterm¶
Claim · All claims
Evidence · 1 study · causalr2·q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? On the instructor's own final (100 points), the AI arm outperformed the holdout by 2.6 points (p=.040) among all 86 randomized students, while the pre-treatment midterm showed no difference between arms. → Shihao Yang 2026
Evidence¶
Shihao Yang 2026¶
Shihao Yang, Marshall Van Alstyne, and Chrysanthos Dellarocas. (2026). When AI Tutors Speak: Evidence from a Randomized Field Experiment. https://arxiv.org/abs/2609.23958 doi:10.48550/arxiv.2609.23958
q3 · i? · causal · r2
Instructor-administered course assessment on file for all 86 randomized students and for 395 classmates who never enrolled, used as an instrument the researchers did not design and on a sample that did not attrit; "2.6 points of 100,p=.040".
"The instructor's own final, on the intact randomization of 86 students, shows the same direction (2.6 points of 100,p=.040), and his pre-treatment midterm shows no difference between arms."
Discussion¶
Related Claims¶
- A quarter of students offered a free AI tutor never tried it once, making first-session adoption a deployment problem — related
- Access to a structured conversational AI tutor increased post-test content scores over a no-AI holdout with ambient consumer AI available — related
- Ambient consumer AI use in the holdout did not close the structured-tutor gap — related
- Evaluating GenAI-generated solutions produced no significant differences on midterm scores, final exam scores, course percentages, or normalized exam change relative to solving problems directly — reports the opposite
- A strategic resource-use exam playbook intervention raised final course grades in introductory statistics across two randomized cohorts, with a dosage effect for completing it twice — related
- Prior GenAI-solution evaluation showed no significant benefit on exam problems structurally aligned with the homework, with the midterm comparison inconclusive due to a floor effect — reports the opposite
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
instructor-final-confirms-tutor-advantage-direction |
title |
The instructor's own final exam showed the same direction of advantage on the intact randomized sample, with no difference on the pre-treatment midterm |
description |
The instructor's own final exam showed the same direction of advantage on the intact randomized sample, with no difference on the pre-treatment midterm |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — shihao-yang-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.