AI TA condition shows no statistically significant differences in perceived task-comprehension support, correctness, or interaction stress¶

Claim · All claims
Evidence · 1 study · causal r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Differences across the four AI TA conditions were not statistically significant for perceived support for task comprehension, provision of correct information, or stressful interaction, though Socratic conditions descriptively rated lower on comprehension support and Socratic + Full Context rated highest on stress. → Madeleine Eastwood 2026

Evidence¶

Madeleine Eastwood 2026¶

Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, and Amanpreet Kapoor. (2026). Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming. https://arxiv.org/abs/2609.29995 doi:10.48550/arxiv.2609.29995

q3 · i? · causal · r2

Post-survey Likert perception measures from the same randomized experiment (N=132); Kruskal-Wallis H tests found no significant condition differences on these three measures. Descriptively, "students in the Direct conditions rated the AI TA higher on task comprehension than those in the Socratic conditions".

"Differences across the four conditions were not statistically sig- nificant for perceived support fortask comprehension( 𝜒 2(3)= 4.79, 𝑝=. 188), provision ofcorrect information( 𝜒 2(3)= 1.71, 𝑝=. 635), orstressful interaction( 𝜒 2(3)= 4.50, 𝑝=. 213)."

Discussion¶

Learner Variables¶

• Affect Regulation — outcome: instruction changes it

Related Claims¶

• Students rate the Socratic + Full Context AI TA significantly lowest for perceived support of task completion — related
• Direct conditions descriptively show higher post-task full comprehension than Socratic conditions, with Direct + Full Context highest and Socratic + Full Context lowest — related
• The effect of context awareness on perceived task-completion support depends on the pedagogical guidance style it is paired with — related
• Students rate a Socratic AI chatbot highest for knowledge-based skills (median ≈ 4.0/5) and positively for overall effectiveness (median ≈ 3.4/5) in an introductory mechanics course — related
• Students perceived diverse and generic explanations as similarly helpful, with no statistically significant difference across explanation-sequence groups — related
• 15% of students used an external LLM, with the highest descriptive rate in the Socratic + Full Context condition, mainly citing inadequate guidance — related
• Socratic + No Context produces the highest query volume and Direct + No Context the longest messages — related
• Socratic feedback is associated with higher task orientation and comprehension monitoring than directive feedback; on feedback regulation it has no main effect, but its effect depends on agent type — related