Evaluating GenAI-generated solutions produced no significant differences on midterm scores, final exam scores, course percentages, or normalized exam change relative to solving problems directly¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q3
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 2 of 2 entries) · not reviewed by a person

Subclaims¶

q3 i? In a randomized crossover algorithms course (N=220), midterm scores did not differ significantly between the solving and GenAI-evaluation groups (t(218)=-0.58, p=.56, d=-0.08). → Ethan Dickey 2026 q3 i? Final exam scores (t(218)=-0.39, p=.70, d=-0.05), normalized change (d=0.03), overall course percentages (d=0.13), and letter-grade distributions (Cramer's V=.07) were all statistically indistinguishable between groups. → Ethan Dickey 2026 (2)

Evidence¶

Ethan Dickey 2026¶

Ethan Dickey, Marios Mertzanidis, Alexandros Psomas. (2026). Is Solving Better Than Evaluating GenAI Solutions? https://arxiv.org/abs/2607.27586 doi:10.48550/arxiv.2607.27586

q3 · i? · causal · r2

Randomized A/B crossover study in a junior-level algorithms course; independent-samples t-tests after assumption checks. Midterm showed "a negligible effect size (Cohen's𝑑=− 0.08)" with CI [-4.8, 2.6], indicating no significant group difference.

"Midterm examination scores did not differ significantly between groups,𝑡( 218)=− 0.58,𝑝=. 56, with a 95% CI of[−4.8, 2.6] points and a negligible effect size (Cohen's𝑑=− 0.08)."

Ethan Dickey 2026 (2)¶

Ethan Dickey, Marios Mertzanidis, Alexandros Psomas. (2026). Is Solving Better Than Evaluating GenAI Solutions? https://arxiv.org/abs/2607.27586

q3 · i? · causal · r2

Final exam (maximum 113) comparison in the same crossover study; the article also reports no significant differences for normalized change (d=0.03), course percentages (d=0.13), and letter grades (chi-squared p=.38, Cramer's V=.07).

"Final examination scores likewise showed no significant difference,𝑡( 218)=− 0.39, 𝑝=.70, 95% CI[−5.4,3.6]points,𝑑=−0.05."

Discussion¶

Related Claims¶

• Prior GenAI-solution evaluation showed no significant benefit on exam problems structurally aligned with the homework, with the midterm comparison inconclusive due to a floor effect — a narrower finding that bears on this claim
• Midterm self-efficacy (comfort and confidence) did not differ between conditions after three assignments, though both varied significantly across algorithmic topics — related
• Students received significantly higher homework scores during GenAI-evaluation periods, but this advantage did not transfer to downstream exams — related
• The instructor's own final exam showed the same direction of advantage on the intact randomized sample, with no difference on the pre-treatment midterm — reports the opposite