Students received significantly higher homework scores during GenAI-evaluation periods, but this advantage did not transfer to downstream exams¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q3 · i1 small
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i1 Within-student difference scores on the GenAI-graded problem differed significantly between groups (Mann-Whitney U=7,046, p=.035, r=.14; t(218)=2.90, p=.004, d=.39). → Ethan Dickey 2026 q3 i1 A parallel significant difference appeared on identical student-solved problems (U=7,119, p=.024, r=.15), which the authors attribute to time-varying factors rather than the intervention. → Ethan Dickey 2026 (2)

Evidence¶

Ethan Dickey 2026¶

Ethan Dickey, Marios Mertzanidis, Alexandros Psomas. (2026). Is Solving Better Than Evaluating GenAI Solutions? https://arxiv.org/abs/2607.27586 doi:10.48550/arxiv.2607.27586

q3 · i1 · causal · r2

Within-student change scores in homework performance across course halves in the crossover study; Group A showed M=18±25 vs Group B 9±23 on GenAI questions, "a small effect size (Cohen's𝑑=.39)".

"For the GenAI-graded problems, the Mann–Whitney test indicated a statistically significant difference between groups (𝑈= 7,046,𝑝=. 035), with a small effect size (𝑟=. 14). A confirmatory independent-samples 𝑡-test yielded a consistent result,𝑡( 218)= 2.90,𝑝=. 004, with a small effect size (Cohen's𝑑=.39)."

Ethan Dickey 2026 (2)¶

Ethan Dickey, Marios Mertzanidis, Alexandros Psomas. (2026). Is Solving Better Than Evaluating GenAI Solutions? https://arxiv.org/abs/2607.27586

q3 · i1 · causal · r2

Identical non-GenAI homework items also differed significantly across halves (r=.15), so the article concludes this pattern means homework-level differences should not be read as direct evidence of a learning effect from the GenAI activity.

"For the student-solved (non-GenAI) problems, the Mann–Whitney test likewise indicated a statistically significant difference between groups (𝑈= 7,119,𝑝=. 024), with a small effect size ( 𝑟=.15)."

Studies bearing on this claim¶

Do gains learners make with generative AI on coursework show up on midterm and final exams taken without it? This table gathers the studies behind this claim and the 8 narrower claims that bear on it: 5 distinct studies (3 associational, 2 causal), q2–q3, 2 of 5 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• AI-supported instruction has a moderate positive effect on student achievement in STEM education (g = 0.670 under a random effects model) — the general claim this finding bears on, with the studies behind it

• Timely feedback on homework increases learning, and computer platforms providing immediate feedback help teachers adapt — the general claim this finding bears on, with the studies behind it

• Evaluating GenAI-generated solutions produced no significant differences on midterm scores, final exam scores, course percentages, or normalized exam change relative to solving problems directly — related
• Prior GenAI-solution evaluation showed no significant benefit on exam problems structurally aligned with the homework, with the midterm comparison inconclusive due to a floor effect — a narrower finding that bears on this claim
• Falsifiable hypothesis: unrestricted AI use on deep-processing tasks will produce a product–process dissociation—higher-rated assignments but lower unaided delayed transfer — a broader claim this one bears on
• Most students reported no change in study habits, but students who adapted their study strategies rated GenAI-evaluation assignments as significantly more helpful — related
• Unrestricted ChatGPT access yields better practice performance but significantly worse exam scores — related
• No significant group differences in knowledge gain or knowledge transfer despite ChatGPT's task-performance advantage — related
• Exam environment critically moderates GenAI learning effects: significant large gains when GenAI is permitted during assessment (g = 0.76) versus non-significant slightly negative effects when it is not (g = -0.06). — related
• More GenAI-susceptible courses show no raised grade floor after ChatGPT: passing-margin and failing/withdrawal probabilities are null under both COVID assumptions — related
• Average homework H-scores correlate negatively with proctored exam performance, with the correlation strengthening from Spring 2024 to Spring 2026 — related
• ITS are effective across instructional roles, whether used as principal means of instruction, integrated class instruction, separate in-class activities, supplementary after-class instruction, or homework — related
• Learning-by-teaching an LLM improved assignment and project performance versus historical cohorts, but not exam performance, in an undergraduate CS course — related
• Across three physics assessments, students who declared GenAI use scored slightly higher than non-users, but none of the differences were statistically significant — related
• AI chatbots provide students detailed feedback, step-by-step homework solutions, and self-quizzing support — related