AI tutor benchmark datasets show essentially no correlation between chatbot scaffolding and student uptake¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Within the AI Tutor Benchmarks category there is essentially no correlation between the two metrics, attributed to restricted variance in uptake under controlled conditions. → Neagu 2026

Evidence¶

Neagu 2026¶

Neagu, A., Wong, J. T. H., Messer, M., Nelson, R., & Johnson, P. B. (2026). Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments. Pluralistic Alignment Workshop @ ICML 2026. https://arxiv.org/abs/2606.15766 doi:10.48550/arxiv.2606.15766

q2 · i? · associational · r2

Category-level Spearman analysis in Section 4.2 over the four benchmark datasets (MathDial, MathTutorBench, QATD2k, PATS). The paper explains the lack of correlation may stem from controlled construction that "assume[s] high student uptake throughout their chats, producing too little variance in uptake to support a meaningful correlation".

"In contrast, inAI Tutor Benchmarksthere is essentially no correlation between the two metrics."

Discussion¶

Related Claims¶

• Interaction patterns of benchmark datasets and real-world deployment categories are statistically distinct in the scaffolding-uptake space — related
• Students in real-world chatbot deployments exhibit lower scaffolding uptake than benchmarks assume, frequently bypassing the chatbot's pedagogical framing — related
• Student uptake is only weakly and inconsistently correlated with chatbot scaffolding in real-world deployments — related
• The LLM-judge scoring pipeline for the two metrics aligns with human labels and human raters — related