The tutor-access learning gain was concentrated in conceptual depth of written answers rather than multiple-choice scores¶

Claim · All claims
Evidence · 1 study · causal r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? The share of short answers reaching SOLO relational quality or better grew about sixfold in the tutored arm, while multiple-choice scores were near ceiling. → Shihao Yang 2026

Evidence¶

Shihao Yang 2026¶

Shihao Yang, Marshall Van Alstyne, and Chrysanthos Dellarocas. (2026). When AI Tutors Speak: Evidence from a Randomized Field Experiment. https://arxiv.org/abs/2609.23958 doi:10.48550/arxiv.2609.23958

q3 · i? · causal · r2

Authors' summary of the preregistered SOLO depth co-primary outcomes (mean SOLO level and share of answers at SOLO≥3) in the randomized field experiment; the abstract reports the relational share rising from 8% to 49% in the tutored arm against 8% to 27% in the holdout.

"The gain was concentrated in the conceptual depth of written answers, where the share reaching relational quality grew sixfold, rather than in ceilinged multiple-choice scores."

Discussion¶

Related Claims¶

• Multiple-choice state tests are prone to ceiling and floor effects and do not measure higher-order skills well — the general claim this finding bears on, with the studies behind it
• Access to a structured conversational AI tutor increased post-test content scores over a no-AI holdout with ambient consumer AI available — a broader claim this one bears on
• Multiple choice practice items produce significantly higher performance during retrieval practice than short answer items, with applied short answer the most difficult condition — related
• Successful retrieval practice of more difficult items (short answer, applied) yields greater future predicted success than successful retrieval of easier items (multiple choice, factual) — related
• Practice condition (format/depth) showed no significant differences on posttest performance for either practiced or unpracticed concepts — related
• Reviews citing one randomised trial (Bastani et al. 2025) report that unrestricted GPT-4 access in high-school mathematics raised assisted practice performance but left students worse off after withdrawal — related
• Theory-guided five-step prompting of AI interaction improves open-ended explanation performance in construction education review, while unprompted AI access alone does not — related
• Ambient consumer AI use in the holdout did not close the structured-tutor gap — related