Direct conditions descriptively show higher post-task full comprehension than Socratic conditions, with Direct + Full Context highest and Socratic + Full Context lowest¶

Claim · All claims
Evidence · 1 study · causal r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Across 128 non-blank post-task explanations, 57% showed full comprehension; by condition, Direct + Full Context had the highest proportion of full-comprehension responses (67%) and Socratic + Full Context the lowest (48%). → Madeleine Eastwood 2026

Evidence¶

Madeleine Eastwood 2026¶

Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, and Amanpreet Kapoor. (2026). Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming. https://arxiv.org/abs/2609.29995 doi:10.48550/arxiv.2609.29995

q3 · i? · causal · r2

Inductive coding of open-ended post-task explanations of the matrix debugging bug, administered immediately after task completion in the randomized experiment. Overall, 57% showed full comprehension, 30% partial/impure, 11% minimal, and 2% ungaugable; no statistical test of condition differences is reported.

"the Direct + Full Context group had the highest proportion of full-comprehension responses (67%), followed by Direct + No Context (61%), Socratic + No Context (50%), and Socratic + Full Context (48%)."

Discussion¶

Related Claims¶

• In an immediate post-test design, participants with an unrestricted chatbot achieved significantly higher learning deltas than participants with a Socratic chatbot (d = 0.80) or an EEG-adaptive exercise system (d = 0.85) — the general claim this finding bears on, with the studies behind it
• AI TA condition shows no statistically significant differences in perceived task-comprehension support, correctness, or interaction stress — related
• Socratic + No Context produces the highest query volume and Direct + No Context the longest messages — related
• Students rate the Socratic + Full Context AI TA significantly lowest for perceived support of task completion — related
• 15% of students used an external LLM, with the highest descriptive rate in the Socratic + Full Context condition, mainly citing inadequate guidance — related
• The effect of context awareness on perceived task-completion support depends on the pedagogical guidance style it is paired with — related
• Students rate a Socratic AI chatbot highest for knowledge-based skills (median ≈ 4.0/5) and positively for overall effectiveness (median ≈ 3.4/5) in an introductory mechanics course — related