Exam environment critically moderates GenAI learning effects: significant large gains when GenAI is permitted during assessment (g = 0.76) versus non-significant slightly negative effects when it is not (g = -0.06).¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q3 · i2 medium
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i2 The exam environment (whether GenAI assistance was permitted during assessment) significantly moderates learning effects (Q_M(1) = 7.23, p = 0.007), accounting for 45.49% of total heterogeneity. → Maier 2026 q3 i2 When AI tools were permitted during the assessment, the estimated effect was large and statistically significant (g = 0.76, 95% CI [0.24, 1.28], p = 0.004); when not allowed, the effect was non-significant and slightly negative (g = -0.06, 95% CI [-0.36, 0.24], p = 0.674). → Maier 2026

Evidence¶

Maier 2026¶

Maier, S., Gunzenhäuser, M., Schweisthal, J., Schneider, M., & Feuerriegel, S. (2026). A meta-analysis of the effect of generative AI on productivity and learning in programming. arXiv preprint. https://arxiv.org/abs/2605.04779 doi:10.48550/arxiv.2605.04779

q3 · i2 · quant-synthesis · r2

Univariate mixed-effects meta-regression across 11 learning effect sizes testing exam environment as moderator. The article reports the omnibus test significant (Q_M(1) = 7.23, p = 0.007, 45.49% of heterogeneity). "when AI tools were permitted during the assessment, the estimated effect was large and statistically significant (g=0.76,95%CI: [0.24, 1.28],p= 0.004)"; without GenAI access, g = -0.06 (95% CI [-0.36, 0.24], p = 0.674).

"In contrast, when AI tools were permitted during the assessment, the estimated effect was large and statistically significant (g=0.76,95%CI: [0.24, 1.28],p= 0.004)."

Discussion¶

Related Claims¶

• Across three physics assessments, students who declared GenAI use scored slightly higher than non-users, but none of the differences were statistically significant — related
• GenAI-assisted education produces significantly higher higher-order thinking abilities than non-GenAI approaches (g = 0.72) — a broader claim this one bears on
• Study setting significantly moderates GenAI productivity effects: laboratory experiments show large significant gains (g = 0.73) while enterprise (g = 0.19) and open-source (g = 0.01) effects are substantially smaller and non-significant. — related
• GenAI-assisted education produces significantly higher academic achievement than non-GenAI-assisted approaches (g = 0.40) — a broader claim this one bears on
• GenAI assistance shows no statistically significant pooled effect on programming learning outcomes (g = 0.14, 95% CI (-0.18, 0.47)), with substantial heterogeneity. — related
• Students received significantly higher homework scores during GenAI-evaluation periods, but this advantage did not transfer to downstream exams — related
• GenAI coding assistance produces a statistically significant moderate positive effect on developer productivity (g = 0.33), with substantial heterogeneity across settings. — related
• GenAI-assisted education produces significantly higher learning motivation than non-GenAI approaches (g = 0.81) — related