Repeated opportunities to apply tutor moves did not by themselves improve execution: opportunity count, lesson completion, and their interaction showed no significant effects on successful execution¶

Claim · All claims
Evidence · 1 study · causal r1 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? A generalized linear mixed-effects growth model found no statistically significant effect of cumulative opportunity count on successful execution (β=0.06, p=.39), a positive but non-significant lesson-completion effect (β=0.35, p=.098), and a non-significant interaction (β=-0.07, p=.36). → Thomas 2026

Evidence¶

Thomas 2026¶

Thomas, D. R., Kamikazi, M. C. A., Brandt, C., Borchers, C., & Koedinger, K. R. (2026). AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice. https://arxiv.org/abs/2606.18617 doi:10.48550/arxiv.2606.18617

q3 · i? · causal · r1

Generalized linear mixed-effects growth curve model predicting probability of successful execution conditional on opportunity, with cumulative opportunity count, lesson completion, and their interaction as fixed effects and a random intercept for tutors. No effect sizes are printed.

"Results indicated no statistically significant effect of opportunity count on successful execution (β= 0.06,p=.39), suggesting limited evidence of improvement purely through repeated opportunities."

Discussion¶

Related Claims¶

• After training, tutors were significantly more likely to encounter pedagogical opportunities (61.1% to 68.9%) and showed higher execution quality within those opportunities (65.5% to 68.1%) — related
• Interrupted time series analysis found tutor quality improved gradually over time (β=0.01, p=.022) with no immediate level change or slope change at the training intervention — related
• Tutor training performance predicts real-life tutoring transcript scores, with a one SD increase in lesson performance associated with a 0.25 SD increase in transcript scores — related
• Lasting pedagogical change is more likely when teachers try out strategies, receive feedback, and iteratively improve across multiple workshops — a broader claim this one bears on
• Performance during the virtual pipetting lesson predicts real-life transfer test performance — reports the opposite