Study setting significantly moderates GenAI productivity effects: laboratory experiments show large significant gains (g = 0.73) while enterprise (g = 0.19) and open-source (g = 0.01) effects are substantially smaller and non-significant.¶

Claim · All claims
Evidence · 1 study · quant-synthesis r3 · q3 · i2 medium
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i2 Study setting significantly moderates the productivity effect of GenAI (Q_M(2) = 10.51, p = 0.005), accounting for approximately 36% of heterogeneity across studies. → Maier 2026 q3 i2 Laboratory experiments yield a large and significant effect (g = 0.73, p < 0.001), whereas enterprise (g = 0.19, p = 0.448) and open-source settings (g = 0.01, p = 0.975) are substantially smaller and do not differ significantly from zero. → Maier 2026

Evidence¶

Maier 2026¶

Maier, S., Gunzenhäuser, M., Schweisthal, J., Schneider, M., & Feuerriegel, S. (2026). A meta-analysis of the effect of generative AI on productivity and learning in programming. arXiv preprint. https://arxiv.org/abs/2605.04779 doi:10.48550/arxiv.2605.04779

q3 · i2 · quant-synthesis · r3

Univariate mixed-effects meta-regression across 16 productivity effect sizes comparing laboratory, enterprise, and open-source settings. The article reports the omnibus test significant (Q_M(2) = 10.51, p = 0.005, ~36% of heterogeneity) and subgroup estimates: "Laboratory experiments yield a large and significant effect (g= 0.73,p <0.001)" versus non-significant enterprise and open-source effects.

"Laboratory experiments yield a large and significant effect (g= 0.73,p <0.001), whereas effects in enterprise (g= 0.19,p= 0.448) and open-source settings (g= 0.01,p= 0.975) are substantially smaller and do not differ signif- icantly from zero."

Discussion¶

Related Claims¶

• Variation in meta-analytic effect sizes stems from synthesists' methodological choices (inclusion criteria, data extraction, formulas, outlier handling) and from study quality — the general claim this finding bears on, with the studies behind it
• Exam environment critically moderates GenAI learning effects: significant large gains when GenAI is permitted during assessment (g = 0.76) versus non-significant slightly negative effects when it is not (g = -0.06). — related
• GenAI coding assistance produces a statistically significant moderate positive effect on developer productivity (g = 0.33), with substantial heterogeneity across settings. — related
• Instructional model shows a significant subgroup difference (traditional classroom g = 1.19 vs. AI teaching g = 0.69) that does not remain robust in multivariate meta-regression — related
• Baseline capability moderates AI productivity benefits non-monotonically across AI's capability frontier — related
• Study design moderates the technology effect: between-subject studies yield a significantly higher average effect than within-subject studies — related
• GenAI assistance shows no statistically significant pooled effect on programming learning outcomes (g = 0.14, 95% CI (-0.18, 0.47)), with substantial heterogeneity. — related