In Study 1, ChatTutor students answered significantly more new follow-up conceptual knowledge items correctly than teaching-as-usual students, controlling for the testing effect¶

Claim · All claims
Evidence · 1 study · causal r3 · q3 · i2 medium
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i2 Learning condition significantly affected performance on five new high-difficulty conceptual knowledge items administered only at follow-up, η2 = 0.070, with ChatTutor exceeding teaching-as-usual (p = 0.010) but not ChatGPT (p = 0.957). → Guido Makransky 2025

Evidence¶

Guido Makransky 2025¶

Guido Makransky, Ban M. Shiwalia, Tue Herlau, Steven Blurton. (2025). Beyond the "Wow" Factor: Using Generative AI for Increasing Generative Sense-Making. Educational Psychology Review. https://doi.org/10.1007/s10648-025-10039-x

q3 · i2 · causal · r3

In the pre-registered Study 1 experiment, five new high-difficulty items were added at follow-up "to ensure that we could measure conceptual knowledge while accounting for testing effect"; the ANOVA gave "F(2, 121) = 4.551, p = 0.012, η2 = 0.070" and post hocs showed ChatTutor (M = 2.58) above teaching-as-usual (M = 1.87), p = 0.010, but not above ChatGPT (M = 2.56), p = 0.957.

"The one-way ANOVA showed a significant effect of learning condition on performance on the five new conceptual knowledge questions, F(2, 121) = 4.551, p = 0.012, η2 = 0.070."

Discussion¶

Related Claims¶

• In Study 1, university students using ChatTutor scored significantly higher than teaching-as-usual students on a delayed conceptual knowledge test four weeks after the lesson, but did not differ significantly from ChatGPT students — related
• In Study 1, the three conditions did not differ significantly on the immediate post-test of conceptual knowledge — related
• In Study 1, ChatTutor produced significantly higher trust, enjoyment, and behavioral intentions than ChatGPT, while students were blind to condition — related