Cumulative rapid experimentation improved next-item correctness by 10% and cognitive engagement by 14% over five months¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Across over 40 experiments in the five months before the paper, most yielding small gains, cumulative improvements reached next-item correctness by 10% and cognitive engagement by 14%. → Udeshi 2026

Evidence¶

Udeshi 2026¶

Udeshi, T., Khazenzon, A., Khan, K., Breen, N., Corwin, R., DiGiano, C., Weatherholtz, K., Zaluski, M. (2026). Methodologies for Improving the Quality of AI Tutoring in K-12 Education. https://sites.google.com/khanacademy.org/aied2026supplementalmaterial/home doi:10.1007/978-3-032-29755-6_14

q2 · i? · causal · r2

The authors' retrospective summary of their experimentation program, reported in the discussion section rather than as a single controlled experiment; the figures are aggregate platform-level changes.

"in the 5 months leading up to this paper we conducted over 40 experiments. Most experiments yield small gains but cumulatively the improvements have been significant: next-item correctness by 10%, cognitive engagement by 14%."

Discussion¶

Related Claims¶

• Model selection experiments show newer models are not strict improvements, with component-specific effects on quality metrics — related
• Platform-enabled experimentation research draws on multiple intellectual lineages, including intelligent tutoring systems, formative feedback, and exemplar platforms like ASSISTments — related
• Cumulative meta-analysis shows reported GenAI productivity effects remained stable over time (stabilizing at g = 0.331) despite rapid advancement of underlying foundation models. — related
• Providing the tutor with student mastery and practice-history context improved engagement and next-item correctness — a narrower finding that bears on this claim