Model selection experiments show newer models are not strict improvements, with component-specific effects on quality metrics¶

Claim · All claims
Evidence · 1 study (3 entries) · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Migrating the Exercise main completion from GPT-4o to GPT-4.1 decreased tutor giving away the final answer by 96.66% ± 4.66% and increased behavioral engagement by 9.93% ± 2.11% in a do-no-harm test. → Udeshi 2026 q2 i? Migrating the Exercise Math Agent from GPT-4o to GPT-4.1 increased next-item correctness by 3.41% ± 3.41% and cognitive engagement by 19.29% ± 12.09% but decreased behavioral engagement by 3.54% ± 1.80%. → Udeshi 2026 (2) q2 i? Migrating the Tutor Me input classifier from GPT-4.1 to GPT-4.1-mini increased cognitive engagement by 11.79% ± 8.52%, decreased time-to-first-token latency (P50) by 14.40% ± 1.30%, and increased procedural-problem classifications by 19.10% ± 4.90% without decreasing the proportion successfully solved. → Udeshi 2026 (3)

Evidence¶

Udeshi 2026¶

Udeshi, T., Khazenzon, A., Khan, K., Breen, N., Corwin, R., DiGiano, C., Weatherholtz, K., Zaluski, M. (2026). Methodologies for Improving the Quality of AI Tutoring in K-12 Education. https://sites.google.com/khanacademy.org/aied2026supplementalmaterial/home doi:10.1007/978-3-032-29755-6_14

q2 · i? · causal · r2

Live experiment comparing AI tutor performance when the main Exercise completion was generated with GPT-4o versus GPT-4.1, run as a do-no-harm test that would launch if the newer model matched the incumbent's performance.

"Tutor giving away final answer decreased by 96.66% ± 4.66% ● Behavioral engagement increased by 9.93% ± 2.11%"

Udeshi 2026 (2)¶

Udeshi, T., Khazenzon, A., Khan, K., Breen, N., Corwin, R., DiGiano, C., Weatherholtz, K., Zaluski, M. (2026). Methodologies for Improving the Quality of AI Tutoring in K-12 Education. https://sites.google.com/khanacademy.org/aied2026supplementalmaterial/home. doi:10.1007/978-3-032-29755-6_14

q2 · i? · causal · r2

Live experiment comparing Math Agent completions generated with GPT-4o versus GPT-4.1, showing mixed effects across the three metrics for this component.

"Next item correctness increased by 3.41% ± 3.41% ● Cognitive engagement increased by 19.29% ± 12.09% ● Behavioral engagement decreased by 3.54% ± 1.80%"

Udeshi 2026 (3)¶

Udeshi, T., Khazenzon, A., Khan, K., Breen, N., Corwin, R., DiGiano, C., Weatherholtz, K., Zaluski, M. (2026). Methodologies for Improving the Quality of AI Tutoring in K-12 Education. https://sites.google.com/khanacademy.org/aied2026supplementalmaterial/home. doi:10.1007/978-3-032-29755-6_14

q2 · i? · causal · r2

After building an offline evaluation dataset for the Tutor Me request classifier and observing higher-than-expected accuracy, the team tested a more compact model for lower computational impact and latency.

"Cognitive engagement increased by 11.79% ± 8.52% ● Time to first token latency (P50) decreased by 14.40% ± 1.30%"

Discussion¶

Related Claims¶

• Limiting verbose Math Agent guidance sharply reduced answer giveaway but decreased cognitive engagement — related
• Providing the tutor with student mastery and practice-history context improved engagement and next-item correctness — related
• Cross-model validation shows consistent degradation under multi-strategy attack across Llama-3.3-70B, GPT-4o-mini, and Claude-3.5-Haiku, with significant differences between models — related
• GPT-4o mini showed progressive turn-level convergence with accumulating context while larger models showed increasing or stable error — related
• Cumulative rapid experimentation improved next-item correctness by 10% and cognitive engagement by 14% over five months — related
• Hint-not-solution is the most challenging rubric dimension, with scores ranging from 6% to 81% across models — related
• Under FSM metrics, GPT-3.5-Turbo outperforms GPT-4o-Mini, reversing that model's higher rank on coding-oriented leaderboards — a narrower finding that bears on this claim
• Gemini Flash 2.0 was selected for production over GPT-4o-mini and Llama 3.1 for its superior faithfulness and speed-to-cost ratio — related
• Including agent role descriptions improves the classroom Manager Agent's decision precision — related
• The adapted 4B model outperforms zero-shot cloud baselines on lexical and semantic measures despite lower raw accuracy — related
• SET-NLP shows expansion of the methodological repertoire without simple replacement of earlier approaches by newer ones — related
• Feedback effectiveness varied by content recency: guidance-heavy feedback performed less consistently on newer material released after the model's knowledge cutoff — related
• Newer LLM generations such as GPT-4 generally outperform GPT-3.5 in short-answer grading accuracy, but this advantage is not consistent across all tasks and domains — related
• Human-written AGENTS.md context files improve agent performance by approximately 4 per cent, while LLM-generated context files decrease performance by 3 per cent and increase token cost by over 20 per cent, as the article reports citing recent research — related
• A git-history audit found 96.3% of AI-generated answer-key question blocks needed no post-release semantic repair — related
• Gemma and LLaMA produce higher-quality responses than GPT-2 by TIGERSCORE, with accuracy and comprehension gaps remaining — related
• In a 10-student pilot, EduGuard produced higher post-test accuracy, larger normalized learning gain, lower calibration error, and lower overreliance than GPT-4o-mini Tutor — related