Prior GenAI-solution evaluation showed no significant benefit on exam problems structurally aligned with the homework, with the midterm comparison inconclusive due to a floor effect¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? On the midterm aligned question there was no significant difference (U=5,644.5, p=.30, r=-0.058), but a pronounced floor effect (median = 0 in both groups) limits sensitivity. → Ethan Dickey 2026 q3 i? On the final aligned exam question no significant difference was observed (U=6,191.5, p=.76, r=.020), carrying the transfer comparison. → Ethan Dickey 2026 (2)

Evidence¶

Ethan Dickey 2026¶

Ethan Dickey, Marios Mertzanidis, Alexandros Psomas. (2026). Is Solving Better Than Evaluating GenAI Solutions? https://arxiv.org/abs/2607.27586 doi:10.48550/arxiv.2607.27586

q3 · i? · causal · r2

Mann-Whitney analysis of exam items out of 15 points, each aligned with a prior homework exercise in the crossover study; midterm inferential result was U=5,644.5, p=.30, with "median = 0 in both groups".

"The midterm item's pronounced floor effect (median = 0 in both groups) limits its sensitivity to detect group differences; the transfer comparison is therefore carried primarily by the final-exam item, and conclusions about midterm transfer should be regarded as inconclusive rather than null."

Ethan Dickey 2026 (2)¶

Ethan Dickey, Marios Mertzanidis, Alexandros Psomas. (2026). Is Solving Better Than Evaluating GenAI Solutions? https://arxiv.org/abs/2607.27586

q3 · i? · causal · r2

Final-exam aligned item in the same crossover design, where group roles had reversed; the KS robustness check corroborated (D=.073, p=.90), so prior GenAI-evaluation participation showed no detectable transfer advantage.

"Similarly, for thefinal, no statistically signifi- cant difference was observed on the final exam item (𝑈= 6,191.5, 𝑝=. 76), with a negligible effect size (𝑟=. 020)."

Discussion¶

Related Claims¶

• Evaluating GenAI-generated solutions produced no significant differences on midterm scores, final exam scores, course percentages, or normalized exam change relative to solving problems directly — a broader claim this one bears on
• Midterm self-efficacy (comfort and confidence) did not differ between conditions after three assignments, though both varied significantly across algorithmic topics — related
• Students received significantly higher homework scores during GenAI-evaluation periods, but this advantage did not transfer to downstream exams — a broader claim this one bears on
• The instructor's own final exam showed the same direction of advantage on the intact randomized sample, with no difference on the pre-treatment midterm — reports the opposite