15% of students used an external LLM, with the highest descriptive rate in the Socratic + Full Context condition, mainly citing inadequate guidance¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Overall 15% of students (20 of 132) reported using another LLM or chatbot; rates differed descriptively but not significantly across conditions, with Socratic + Full Context highest at 23% (7 of 30). → Madeleine Eastwood 2026 q3 i? Students' open-ended reasons for using another LLM fell into three categories: inadequate guidance (n=17), going in circles (n=4), and forgetting context (n=2). → Madeleine Eastwood 2026 (2)

Evidence¶

Madeleine Eastwood 2026¶

Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, and Amanpreet Kapoor. (2026). Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming. https://arxiv.org/abs/2609.29995 doi:10.48550/arxiv.2609.29995

q3 · i? · causal · r2

Self-reported external LLM use from the post-survey in the randomized experiment; a chi-square test found no significant condition differences. The Socratic + Full Context condition reported the highest rate (23%, 7 of 30 students) versus 9–15% in the other conditions.

"Overall, 15% of students (20 of 132) re- ported using another LLM or chatbot during the experiment. Although usage rates differed descriptively across conditions, the differences were not statistically significant (𝜒 2(3)= 2.47, 𝑝=. 48)."

Madeleine Eastwood 2026 (2)¶

Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, and Amanpreet Kapoor. (2026). Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming. https://arxiv.org/abs/2609.29995

q3 · i? · causal · r2

Inductive content analysis of open-ended survey responses from students who reported using another LLM. The most common concern was that the chatbot "did not provide enough support, gave explanations that were too general or inflexible"; one student called it "super rigid".

"Their reasons were abstracted into three not mutu- ally exclusive categories:inadequate guidance( 𝑛= 17),going in circles( 𝑛= 4), andforgetting context( 𝑛= 2)."

Discussion¶

Related Claims¶

• In an immediate post-test design, participants with an unrestricted chatbot achieved significantly higher learning deltas than participants with a Socratic chatbot (d = 0.80) or an EEG-adaptive exercise system (d = 0.85) — the general claim this finding bears on, with the studies behind it
• Socratic + No Context produces the highest query volume and Direct + No Context the longest messages — related
• Direct conditions descriptively show higher post-task full comprehension than Socratic conditions, with Direct + Full Context highest and Socratic + Full Context lowest — related
• Students rate the Socratic + Full Context AI TA significantly lowest for perceived support of task completion — related
• AI TA condition shows no statistically significant differences in perceived task-comprehension support, correctness, or interaction stress — related
• The effect of context awareness on perceived task-completion support depends on the pedagogical guidance style it is paired with — related