GenAI assistance shows no statistically significant pooled effect on programming learning outcomes (g = 0.14, 95% CI [-0.18, 0.47]), with substantial heterogeneity.¶

Claim · All claims
Evidence · 1 study · quant-synthesis r3 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Across 11 effect sizes from 10 studies, the pooled effect of GenAI assistance on learning outcomes is small and statistically non-significant (Hedges' g = 0.14, 95% CI [-0.18, 0.47], SE = 0.17, p = 0.389). → Maier 2026 q3 i? Substantial heterogeneity exists in learning effects (I² = 86%, τ² = 0.25, Q(10) = 54.96, p < 0.001), suggesting effects depend strongly on contextual setting such as exam access to GenAI. → Maier 2026

Evidence¶

Maier 2026¶

Maier, S., Gunzenhäuser, M., Schweisthal, J., Schneider, M., & Feuerriegel, S. (2026). A meta-analysis of the effect of generative AI on productivity and learning in programming. arXiv preprint. https://arxiv.org/abs/2605.04779 doi:10.48550/arxiv.2605.04779

q3 · i? · quant-synthesis · r3

Random-effects meta-analysis of 11 effect sizes from 10 studies comparing exam performance after instruction with versus without GenAI access. The article reports "a small but statistically non-significant pooled effect" (g = 0.14, 95% CI [-0.18, 0.47], SE = 0.17, p = 0.389) with I² = 86% and τ² = 0.25 (Q(10) = 54.96, p < 0.001).

"Overall, we find a small but statistically non-significant pooled effect of GenAI assistance on learning outcomes (Hedges’g= 0.14,95%CI:[−0.18,0.47],SE = 0.17,p= 0.389)."

Discussion¶

Related Claims¶

• GenAI coding assistance produces a statistically significant moderate positive effect on developer productivity (g = 0.33), with substantial heterogeneity across settings. — related
• Exam environment critically moderates GenAI learning effects: significant large gains when GenAI is permitted during assessment (g = 0.76) versus non-significant slightly negative effects when it is not (g = -0.06). — related
• Study setting significantly moderates GenAI productivity effects: laboratory experiments show large significant gains (g = 0.73) while enterprise (g = 0.19) and open-source (g = 0.01) effects are substantially smaller and non-significant. — related