Engagement profiles significantly moderated GATs' immediate performance effects: low- and high-engagement profiles benefited while the mid-engagement profile showed a small decrement¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · no dated study recorded · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Engagement profiles significantly moderated immediate learning performance (p=.005, padj=.024), with GATs benefiting low- and high-engagement profiles and a small performance decrement for the mid-engagement profile. → profile-moderation

Evidence¶

profile-moderation¶

Yuri Noviello, Naaz Sibia, Anastasiia Birillo, Thomas Overklift Vaupel Klein, Michael Liut, and Gosia Migut. 2026. AI-Generated Traces for Novice Programmers: Learning Effects and Learner Differences in a Multi-Institutional Study. Proceedings of the 31st ACM Conference on Innovation and Technology in Computer Science Education V. 1 (ITiCSE 2026). https://doi.org/10.1145/3803400.3809346

q2 · i? · causal · r2

Exploratory k-means clustering (k=3) on standardized CAP engagement dimensions at UofT only, due to the limited TU Delft sample (N=16); profiles differed primarily in overall engagement magnitude. No moderation was found for exam outcomes, cognitive load, frustration, or situational interest.

"In contrast, pro- files significantly moderated immediate learning performances (𝑝=. 005, 𝑝𝑎𝑑 𝑗 =. 024). Specifically,GATsbenefited low- and high-engagement profiles, while mid-engagement profile showed a small performance decrement relative to textual explanations (Figure 4(b))."

Discussion¶

Learner Variables¶

• Motivation — moderator: an instructional effect differs with it

Related Claims¶

• In the Java-based CS1 course at TU Delft, GATs significantly improved immediate post-intervention performance pooled across topics, driven mainly by the While-loop intervention — related
• In the Python-based CS1 course at UofT, GATs produced no significant overall or topic-specific immediate performance effects — related
• At UofT, GATs produced a reliable end-of-course increase in Constructive engagement, with no effects on Passive, Active engagement, or MSLQ elaboration — related
• GATs produced no reliable immediate learning-experience effects on frustration, cognitive load, or situational interest at either institution — related
• Across both institutions, GATs showed no significant effect on long-term summative exam performance — related
• Moderate learner agency in game-based learning yields better outcomes than no agency or high agency — related
• Teachers' initial SRL competence profiles moderate training effectiveness: initially high-competence teachers benefit more from training — related
• High Overall Engagement and Metacognitive-but-not-Constructive profiles show the highest immediate and delayed knowledge, with metacognitive engagement marking the key difference from lower performers — related