Students rate the Socratic + Full Context AI TA significantly lowest for perceived support of task completion¶

Claim · All claims
Evidence · 1 study · causal r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? In a 2x2 randomized experiment (N=132), condition significantly affected perceived support for task completion, with the Socratic + Full Context AI TA rated lowest (mean rank 48.63, μ=3.53) versus the other three conditions. → Madeleine Eastwood 2026

Evidence¶

Madeleine Eastwood 2026¶

Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, and Amanpreet Kapoor. (2026). Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming. https://arxiv.org/abs/2609.29995 doi:10.48550/arxiv.2609.29995

q3 · i? · causal · r2

Post-survey perception data from the 2x2 randomized controlled experiment in an introductory Python course; Kruskal-Wallis H test across the four AI TA conditions found a significant effect on task-completion support, "𝜒 2(3)= 12.14, 𝑝=. 007". The Socratic + Full Context condition rated lowest (mean rank 48.63, μ=3.53) versus Direct + Full Context (76.85, μ=4.27).

"A Kruskal–Wallis H test revealed a statistically significant effect of condition on perceived support of the AI TA fortask completion (𝜒 2(3)= 12.14, 𝑝=. 007, 𝑁= 132)."

Discussion¶

Related Claims¶

• In an immediate post-test design, participants with an unrestricted chatbot achieved significantly higher learning deltas than participants with a Socratic chatbot (d = 0.80) or an EEG-adaptive exercise system (d = 0.85) — the general claim this finding bears on, with the studies behind it
• AI TA condition shows no statistically significant differences in perceived task-comprehension support, correctness, or interaction stress — related
• The effect of context awareness on perceived task-completion support depends on the pedagogical guidance style it is paired with — related
• Direct conditions descriptively show higher post-task full comprehension than Socratic conditions, with Direct + Full Context highest and Socratic + Full Context lowest — related
• 15% of students used an external LLM, with the highest descriptive rate in the Socratic + Full Context condition, mainly citing inadequate guidance — related
• Students rate a Socratic AI chatbot highest for knowledge-based skills (median ≈ 4.0/5) and positively for overall effectiveness (median ≈ 3.4/5) in an introductory mechanics course — reports the opposite
• Socratic feedback is associated with higher task orientation and comprehension monitoring than directive feedback; on feedback regulation it has no main effect, but its effect depends on agent type — related
• Adult learners perceive conversational AI as thinking partners that support problem exploration without direct instruction — related