Scenario-based tutor lessons produced a significant pooled 7.4% learning gain, driven by three lessons with established construct validity while three newer lessons showed no significant gains¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r1 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Pooled across all lessons, tutors showed a statistically significant 7.4% pretest-to-posttest learning gain (F=27.50, p<.001). → Thomas 2026 q3 i? REACT_ERRORS gained 25.2% (F=30.10, p<.001), DETERMINE_KNOW 14.2% (p=.006), and GIVE_PRAISE 8.1% (p=.009), while AFFIRM_CORRECT, GUIDE_THINKING, and PROMPT_EXPLAIN showed non-significant gains of 1.4% to 2.1% (all p>.05), attributed to a ceiling effect. → Thomas 2026 (2)

Evidence¶

Thomas 2026¶

Thomas, D. R., Kamikazi, M. C. A., Brandt, C., Borchers, C., & Koedinger, K. R. (2026). AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice. https://arxiv.org/abs/2606.18617 doi:10.48550/arxiv.2606.18617

q3 · i? · causal · r1

Univariate ANOVAs on pretest-posttest lesson scores across 433 pooled lesson completions by 86 tutors. The article reports "a statistically significant learning gain of 7.4% (F=27.50,p<.001)"; no effect size is printed.

"Despite the flat growth in these newer lessons, the aggregate analysis on all lessons (pooled) revealed a statistically significant learning gain of 7.4% (F=27.50,p<.001), supporting overall content validity."

Thomas 2026 (2)¶

Thomas, D. R., Kamikazi, M. C. A., Brandt, C., Borchers, C., & Koedinger, K. R. (2026). AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice. https://arxiv.org/abs/2606.18617. doi:10.48550/arxiv.2606.18617

q3 · i? · causal · r1

Per-lesson ANOVAs with time as factor (post vs. pre) on tutors' pretest-posttest scores; Table 4 prints per-lesson means and F-statistics. The three newer lessons had pretest scores already ranging from 0.92 to 0.98 out of 1.0, which the authors attribute to a ceiling effect.

"REACT_ERRORS, achieving a 25.2% gain (F=30.10,p<.001), followed by DETERMINE_KNOW (14.2% gain,p=.006) and GIVE_PRAISE (8.1% gain,p=.009)."

Discussion¶

Related Claims¶

• Problem-based learning shows null or negative effects on basic knowledge acquisition but positive effects on conceptual understanding and transfer relative to direct instruction — the general claim this finding bears on, with the studies behind it
• Peer Tutoring Improves Achievement — a broader claim this one bears on
• AI presents four solution types for overcoming situated learning barriers: adaptive systems, intelligent tutoring in authentic scenarios, administrative automation, and data-driven teacher support — related
• High-dosage tutoring, frequent teacher feedback and coaching, and data-driven instructional practices show moderately strong evidence of association with charter-school achievement impacts — a broader claim this one bears on
• Assignment to BookNook tutoring produces a small but statistically significant increase in reading growth on the NWEA MAP for grade 1-4 students — related
• Among students receiving at least 24 hours of Air Tutors tutoring, no statistically significant long-term effect on statewide math assessments was detected — related
• The Saga blended technology-enhanced tutoring model significantly improves standardized math scores, math GPA, and reduces math course failures (+0.19) — related
• SMART® tutoring produced a statistically significant positive effect on first-graders' word identification after two years of intervention — related
• The best AI tutors exceeded human mean learning gains in Quantitative domains while human tutoring led pooled AI means in all Verbal domains — related
• Pooled AI tutoring produces statistically equivalent GRE learning gains to expert human tutoring — related
• An approximately hour-long interactive computer tutorial on orthographic projection produced significant pre–post performance gains for most students — related