Socratic + No Context produces the highest query volume and Direct + No Context the longest messages¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Query volume differed significantly across conditions (χ²(3)=13.28, p=.004); Socratic + No Context students sent the most queries per problem (μ=11.1) versus 5.1, 4.1, and 4.7 in the other conditions. → Madeleine Eastwood 2026 q3 i? Average message length differed significantly across conditions (χ²(3)=29.96, p<.001); Direct + No Context students wrote the longest messages (μ=126 characters) and Socratic + Full Context the shortest (μ=54). → Madeleine Eastwood 2026 (2)

Evidence¶

Madeleine Eastwood 2026¶

Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, and Amanpreet Kapoor. (2026). Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming. https://arxiv.org/abs/2609.29995 doi:10.48550/arxiv.2609.29995

q3 · i? · causal · r2

Log-data analysis of 2,467 chat messages from the randomized experiment; a Kruskal-Wallis H test found a significant difference in queries across conditions (χ²(3)=13.28, p=.004). The article suggests Socratic guidance without context "required more back-and-forth with the chatbot to make progress".

"Students in the Socratic + No Context condition sent substantially more queries (𝜇= 11.1per problem) than students in the Socratic + Full Context (𝜇= 5.1), Direct + Full Context (𝜇= 4.1), and Direct + No Context (𝜇= 4.7) conditions."

Madeleine Eastwood 2026 (2)¶

Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, and Amanpreet Kapoor. (2026). Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming. https://arxiv.org/abs/2609.29995

q3 · i? · causal · r2

Log-data analysis of average message length across the four conditions; a Kruskal-Wallis H test found a significant difference (χ²(3)=29.96, p<.001). The article suggests that without context, "students often had to describe their situation in more detail before receiving useful support".

"Students in the Direct + No Context condition wrote the longest messages on average (𝜇= 126characters per message), followed by Direct + Full Context (𝜇= 81), Socratic + No Context (𝜇= 69), and Socratic + Full Context ( 𝜇= 54)."

Discussion¶

Related Claims¶

• In an immediate post-test design, participants with an unrestricted chatbot achieved significantly higher learning deltas than participants with a Socratic chatbot (d = 0.80) or an EEG-adaptive exercise system (d = 0.85) — the general claim this finding bears on, with the studies behind it
• Direct conditions descriptively show higher post-task full comprehension than Socratic conditions, with Direct + Full Context highest and Socratic + Full Context lowest — related
• 15% of students used an external LLM, with the highest descriptive rate in the Socratic + Full Context condition, mainly citing inadequate guidance — related
• AI TA condition shows no statistically significant differences in perceived task-comprehension support, correctness, or interaction stress — related
• Student chatbot queries were overwhelmingly task-oriented initial inquiries, dominated by definition requests and pasted assignment questions rather than iterative critique — related
• Socratic-mode participants progressively disengaged from the chatbot: message counts declined after question 5 and EEG engagement fell to its lowest value at question 10 — related