No significant group differences in knowledge gain or knowledge transfer despite ChatGPT's task-performance advantage¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q3 · i0 negligible
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i0 Groups did not differ significantly on pre-test, post-test (knowledge gain) or transfer test scores, with transfer η² = 0.000. → Fan 2025 q3 i0 Knowledge transfer to an AI-in-healthcare test showed no significant differences between the four groups. → Fan 2025 (2)

Evidence¶

Fan 2025¶

Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gasevic, D. (2025). Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance. British Journal of Educational Technology. https://doi.org/10.1111/bjet.13544

q3 · i0 · causal · r2

ANOVA on a 10-item knowledge test on AI in education, pre and post, in the randomised four-group experiment. Neither pre-test nor post-test differed significantly, so knowledge gain did not differ despite the AI group's essay-score advantage.

"The ANOVA results indicated no significant differences between the groups in terms of the pre-test score (F=1.294, p=0.281, η2=0.036) and post-test score (F=0.913, p=0.438, η2=0.030), which means no significant differences in knowledge gain."

Fan 2025 (2)¶

Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gasevic, D. (2025). Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance. British Journal of Educational Technology. https://doi.org/10.1111/bjet.13544

q3 · i0 · causal · r2

ANOVA on a 10-item transfer test on AI in healthcare in the same randomised experiment. The printed effect size was η2=0.000, indicating no detectable group difference in knowledge transfer.

"We compared the transfer test scores between the four groups, and ANOVA results showed that there were no significant differences between the four groups (F=0.019, p=0.996, η2=0.000)."

Discussion¶

Related Claims¶

• Learners supported by ChatGPT, a human expert, checklist tools, or no support show no significant difference in post-task intrinsic motivation — related
• Students received significantly higher homework scores during GenAI-evaluation periods, but this advantage did not transfer to downstream exams — related
• AI multimodal feedback produces post-test learning gains equivalent to educator feedback after controlling for pre-test performance — related
• Project performance improved significantly from pre- to post-feedback across all three feedback conditions with no significant differences between feedback sources — related