The customized agent's advantage in short-term task improvement emerges under directive feedback but not under Socratic feedback¶

Claim · All claims
Evidence · 1 study (4 entries) · causal r1–r2 · q3 · i1–i3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i2 Under directive feedback, the customized agent produced significantly higher gain scores than the general-purpose agent (η2p = 0.123). → Han Xue 2026 (2) q3 i? Under Socratic feedback, there was no significant difference in gain scores between the customized and general-purpose agents. → Han Xue 2026 (3) q3 i3 Gain scores were overall higher under Socratic feedback than under directive feedback (significant main effect of feedback style). → Han Xue 2026 q3 i1 The main effect of agent type on learning outcomes was not statistically significant (η2p = 0.019). → Han Xue 2026 (4)

Evidence¶

Han Xue 2026¶

Han Xue, Cao Jing, Wang Zi, Luo Heng. (2026). The Effects of Agent Type and Feedback Style on Self-Directed Learning: A Mixed-Methods Study. Behavioral Sciences. https://doi.org/10.3390/bs16071069

q3 · i3 · causal · r?

Mixed-design ANOVA on gain scores (revised minus initial draft rubric scores) from 51 postgraduate students across two instructional design tasks. The main effect of feedback style was significant with η2p = 0.237; a significant agent type × feedback style interaction was also found (F(1,49) = 4.807, p = 0.033, η2p = 0.089).

"Mixed-design ANOVA revealed a significant main effect of feedback style on learning outcomes (F(1,49) = 15.218, p < 0.001, η2p = 0.237), with gain scores overall being higher under Socratic feedback than under directive feedback."

Han Xue 2026 (2)¶

Han Xue, Cao Jing, Wang Zi, Luo Heng. (2026). The Effects of Agent Type and Feedback Style on Self-Directed Learning: A Mixed-Methods Study. Behavioral Sciences. https://doi.org/10.3390/bs16071069

q3 · i2 · causal · r2

Simple-effects analysis reported alongside Figure 2 for the same gain-score ANOVA. Under directive feedback the customized agent outperformed the general-purpose agent (η2p = 0.123); the article states gain scores varied by the combination of agent type and feedback style.

"Under directive feedback, however, the customized agent produced significantly higher gain scores than the general-purpose agent (F(1,49) = 6.843, p = 0.012, η2p = 0.123)."

Han Xue 2026 (3)¶

Han Xue, Cao Jing, Wang Zi, Luo Heng. (2026). The Effects of Agent Type and Feedback Style on Self-Directed Learning: A Mixed-Methods Study. Behavioral Sciences. https://doi.org/10.3390/bs16071069

q3 · i? · causal · r1

Simple-effects analysis from the same study: under Socratic feedback the two agent types did not differ significantly in gain scores; equivalence was not tested.

"simple-effects analysis indicated that under Socratic feedback, there was no significant difference in gain scores between the customized agent and the general-purpose agent (F(1,49) = 0.147, p = 0.703, η2p = 0.003)."

Han Xue 2026 (4)¶

Han Xue, Cao Jing, Wang Zi, Luo Heng. (2026). The Effects of Agent Type and Feedback Style on Self-Directed Learning: A Mixed-Methods Study. Behavioral Sciences. https://doi.org/10.3390/bs16071069

q3 · i1 · causal · r?

Omnibus main effect of agent type on gain scores from the same mixed-design ANOVA was not statistically significant, with a small printed effect size (η2p = 0.019).

"The main effect of agent type was not statistically significant (F(1,49) = 0.961, p = 0.332, η2p = 0.019)."

Discussion¶

Related Claims¶

• In an immediate post-test design, participants with an unrestricted chatbot achieved significantly higher learning deltas than participants with a Socratic chatbot (d = 0.80) or an EEG-adaptive exercise system (d = 0.85) — the general claim this finding bears on, with the studies behind it
• Customized AI agents generate feedback with higher accuracy and specificity than general-purpose agents in self-directed instructional design tasks — related
• Socratic feedback is associated with higher task orientation and comprehension monitoring than directive feedback; on feedback regulation it has no main effect, but its effect depends on agent type — related
• Directive feedback is associated with higher cognitive load than Socratic feedback, with no agent-type effects on learning experiences — related
• As cited second-hand in an AI-literacy framework paper: ChatGPT use improved short-term essay scores but triggered metacognitive laziness with no significant knowledge-transfer gains — related