GenAI effects on externally assessed learning are highly heterogeneous and conditional on prior knowledge, scaffolding, verification, and task design, with pooled effects unstable under publication-bias correction¶

Claim · All claims
Evidence · 1 study (2 entries) · review r2–r3 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 2 of 2 entries) · not reviewed by a person

Subclaims¶

q2 i? A STEM meta-analysis found a positive conventional pooled effect on externally assessed cognitive outcomes, but extreme heterogeneity and a publication-bias-adjusted analysis favouring no overall effect. → Qusay H. Mahmoud 2026 q2 i? Systematic reviews of programming education report short-term efficiency gains alongside over-reliance, weak transfer to independent skill, and recurrent implementation challenges. → Qusay H. Mahmoud 2026 (2)

Evidence¶

Qusay H. Mahmoud 2026¶

Qusay H. Mahmoud. (2026). Judgment-Centred Software Engineering Education: A Post-Hype Review and Framework for AI-Augmented Learning. Preprint, September 2026. https://arxiv.org/abs/2609.29473 doi:10.48550/arxiv.2609.29473

q2 · i? · review · r3

The review reports the Boolzen et al. 2026 STEM meta-analysis: a conventional random-effects model found a positive pooled effect, with "heterogeneity was extreme (I 2 = 96 .32%)" and a prediction interval spanning negative to positive effects; a robust Bayesian analysis adjusting for publication bias favoured no overall effect.

"Its conventional random-effects model found a positive pooled effect on externally assessed cognitive outcomes, but heterogeneity was extreme (I 2 = 96 .32%) and the prediction interval included negative, null, and positive effects."

Qusay H. Mahmoud 2026 (2)¶

Qusay H. Mahmoud. (2026). Judgment-Centred Software Engineering Education: A Post-Hype Review and Framework for AI-Augmented Learning. Preprint, September 2026. https://arxiv.org/abs/2609.29473

q2 · i? · review · r2

The review attributes to Kumar et al.'s systematic review of 72 primary studies "robust short-term efficiency gains" and "weak transfer from AI-assisted completion to independent skill"; other 2026 reviews (Katona; Adejumo; Agbo) converge on benefits conditional on pedagogical strategy.

"Kumar, Wongsirichot, and Nanthaamornphong’s systematic review of 72 primary studies reports robust short-term efficiency gains, recurring over-reliance, weak transfer from AI-assisted completion to independent skill, and strong evidence for assessment redesign."

Discussion¶

Related Claims¶

• GenAI-supported writing instruction yields a statistically significant positive pooled effect on L2/EFL writing performance (g = 0.80) with very high heterogeneity — related
• Effect sizes across GenAI studies are highly heterogeneous, requiring random-effects pooling — related
• GenAI-supported learning is associated with higher learning motivation on average (g = 0.764), but with very high between-study heterogeneity (I2 = 93.7%) — related
• Reviewed literature reports risks of overreliance, AI errors in complex tasks, and students' uncritical acceptance of AI-generated outputs — related
• SRL skills and prior knowledge predict revision performance, while SRL predicts higher and prior knowledge lower cognitive load — related
• Boundary conditions on verification and reliance include prior knowledge, learner characteristics, task stakes, verifiability, costs, accountability, AI system configuration, and multidimensional AI literacy — related
• Evidence of publication bias exists, but the pooled video effects are robust to it — related
• The positive GenAI writing effect is robust across dependence-aware, sensitivity, and risk-of-bias specifications, but the wide prediction interval spans near-zero to large effects — related
• Publication bias is detected in the cross-sectional data set, and bias-corrected estimates are highly heterogeneous — related
• Emerging technologies show a small-to-moderate positive overall effect on TEFL outcomes (g = 0.38, corrected to 0.28 for publication bias) — related