Most unrestricted-mode interactions were direct answer retrieval: 52.9% of exercise interactions were copy-paste of the question, and participant answers were linguistically near-identical to chatbot text in 25.3% of cases¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? 52.9% of Mode 1 exercise interactions were classified as copy-paste question, versus 21.8% in the Socratic mode. → Alexandre Clin Deffarges 2026 q2 i? Mean linguistic similarity between participant answers and chatbot messages was 0.374 in Mode 1 versus 0.044 in Mode 2 (p < 10^-18), with 25.3% of Mode-1 final answers nearly identical to chatbot text versus 0% in Mode 2. → Alexandre Clin Deffarges 2026 (2)

Evidence¶

Alexandre Clin Deffarges 2026¶

Alexandre Clin Deffarges, Nataliya Kosmyna, and Pattie Maes. (2026). Socrates went Nuclear: Comparing Interaction Strategies for AI systems in a Learning Context using Brain Sensing. arXiv preprint arXiv:2609.00584. https://arxiv.org/abs/2609.00584 doi:10.48550/arxiv.2609.00584

q2 · i? · causal · r2

Rule-based classification of each discussion into eight behavioral clusters, applied to the 170 exercise attempts in each conversational mode. Copy-paste dominated Mode 1 (52.9%) while Mode 2 spread across ask for definition (28.8%), copy-paste (21.8%), and no discussion (22.4%).

"52.9% of all exercise interactions in Mode 1 were classified ascopy-paste question, and the remaining interactions were mostly ask for definition(22.4%)."

Alexandre Clin Deffarges 2026 (2)¶

Alexandre Clin Deffarges, Nataliya Kosmyna, and Pattie Maes. (2026). Socrates went Nuclear: Comparing Interaction Strategies for AI systems in a Learning Context using Brain Sensing. arXiv preprint arXiv:2609.00584. https://arxiv.org/abs/2609.00584

q2 · i? · causal · r2

Linguistic similarity analysis comparing each final training answer with the chatbot messages of the corresponding exercise. The authors state this confirms Mode 1 answers were heavily derived from the chatbot while Mode 2 answers were formulated by participants themselves.

"Mean similarity between participant answers and chatbot messages was 0.374 in Mode 1 versus 0.044 in Mode 2 (𝑝< 10−18), and 25.3% of Mode-1 final training answers were nearly identical to the chatbot text, against 0% in Mode 2."

Discussion¶

Related Claims¶

• In an immediate post-test design, participants with an unrestricted chatbot achieved significantly higher learning deltas than participants with a Socratic chatbot (d = 0.80) or an EEG-adaptive exercise system (d = 0.85) — related
• Shallow, copy-pasting engagement occurred but was not the dominant mode of GenAI Tutor use, and was lowest during exam preparation — reports the opposite
• Student chatbot queries were overwhelmingly task-oriented initial inquiries, dominated by definition requests and pasted assignment questions rather than iterative critique — related
• Students' GenAI prompts overwhelmingly exact-copy instructions and request information rather than review — related
• Learners engaged extensively with the chatbot, overwhelmingly communicating in Japanese rather than English — related
• Guest-mode usage dominates lightweight help-seeking, raising an equity concern that students without metacognitive support stay at shallow answer lookup — related
• Socratic-mode participants progressively disengaged from the chatbot: message counts declined after question 5 and EEG engagement fell to its lowest value at question 10 — related
• Socratic participants required nearly twice as many attempts to reach the 80% mastery threshold as Full participants (d = -2.01), and 58% of the exercises of Socratic participants who abandoned the dialogue mid-way never reached the threshold — related