Across both institutions, GATs showed no significant effect on long-term summative exam performance¶

Claim · All claims
Evidence · 1 study · causal r2 · q3
Currency · no dated study recorded · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q3 i? ANCOVA models controlling for prior experience showed no significant condition differences on summative exam outcomes at either institution. → exam-null

Evidence¶

exam-null¶

Yuri Noviello, Naaz Sibia, Anastasiia Birillo, Thomas Overklift Vaupel Klein, Michael Liut, and Gosia Migut. 2026. AI-Generated Traces for Novice Programmers: Learning Effects and Learner Differences in a Multi-Institutional Study. Proceedings of the 31st ACM Conference on Innovation and Technology in Computer Science Education V. 1 (ITiCSE 2026). https://doi.org/10.1145/3803400.3809346

q3 · i? · causal · r2

Student-level ANCOVA of final-exam outcomes at both sites (e.g., While-loop exam rows: TU Delft N=135, Δ=0.746, p=0.390; UofT N=893, Δ=0.808, p=0.369); all adjusted p-values exceeded significance thresholds. Equivalence was not tested.

"Across bothTU DelftandUof T, ANCOVA models controlling for prior experience showed no significant condition differences on summative exam outcomes, with all adjusted 𝑝-values exceeding conventional significance thresholds (Table 1, rows 25-29)."

Discussion¶

Related Claims¶

• GATs produced no reliable immediate learning-experience effects on frustration, cognitive load, or situational interest at either institution — related
• In the Python-based CS1 course at UofT, GATs produced no significant overall or topic-specific immediate performance effects — related
• In the Java-based CS1 course at TU Delft, GATs significantly improved immediate post-intervention performance pooled across topics, driven mainly by the While-loop intervention — related
• Engagement profiles significantly moderated GATs' immediate performance effects: low- and high-engagement profiles benefited while the mid-engagement profile showed a small decrement — related
• CS1-CR exam scores showed a slight, statistically insignificant increase compared to prior semesters — related
• At UofT, GATs produced a reliable end-of-course increase in Constructive engagement, with no effects on Passive, Active engagement, or MSLQ elaboration — related