Instruction type predicts post-test performance with a large effect size after controls (linear and quantile regression)¶

Claim · All claims
Evidence · 1 study · causal r2 · q3 · i2 medium
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i2 Controlling for prior proficiency, ChatGPT experience, topic, test version and time on task, AI tutoring shows a large effect on post-test scores (0.63 SD; 0.73–1.3 SD by quantile regression). → Kestin 2025

Evidence¶

Kestin 2025¶

Kestin, G., Miller, K., Klales, A., Milbourne, T. & Ponti, G. (2025). AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting. Scientific Reports 15:17458. https://doi.org/10.1038/s41598-025-97652-6

q3 · i2 · causal · r2

Linear regression model (Table S1) clustered at the student level in the crossover RCT, controlling for pre-test, midterm, FCI, ChatGPT experience, topic, test version and time on task; the "effect size of 0.63" was deemed an underestimation.

"While the linear regression suggests an effect size of 0.63, this is an underestimation due to ceiling effect; a quantile regression allows us to provide an estimate of the effect size that avoids ceiling effect in the post-test scores. Such an analysis provides an effect size in the range of 0.73 to 1.3 standard deviations."

Discussion¶

Related Claims¶

• The AI-tutoring advantage holds across prior-ability and scientific-attitude subpopulations — related
• Students learn more with a research-based AI tutor than with in-class active learning on identical content (post-test scores) — possibly the same claim (merge candidate)
• AI access raises immediate unaided test scores by 7.2 percentage points (0.28 SD) after a timed learning phase — related
• In the Java-based CS1 course at TU Delft, GATs significantly improved immediate post-intervention performance pooled across topics, driven mainly by the While-loop intervention — related
• GLAT scores predicted performance on GenAI-supported tasks better than perceived ChatGPT proficiency — related
• In-class chat engagement significantly predicts higher test scores in MAIC courses — related
• In one RCT, a content-rich prompt-engineered generative AI chat tutor improved performance and engagement in a Harvard physics course — related