Students who used LLMs on assignments scored significantly lower than non-users on the first midterm (large effect size), but this gap narrowed and was non-significant on the later midterms¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r1 · q2 · i2 medium
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i2 No-LLM students significantly outperformed LLM users on Midterm 1 with a large effect size; the difference decreased across the subsequent two midterms, and LLM users improved by about 10% from first to last midterm. → Minju Park 2026 q2 i? The gap between LLM users and non-users decreased over the subsequent two midterms, with the LLM-using group improving by approximately 10% from the first to the last midterm. → Minju Park 2026 (2)

Evidence¶

Minju Park 2026¶

Minju Park, Ivan Orozco Vasquez, and Cristina Conati. (2026). Characterizing Students' LLM Usage Behaviors and Their Association with Learning in Critical Thinking Tasks. https://arxiv.org/abs/2605.04534 doi:10.48550/arxiv.2605.04534

q2 · i2 · associational · r1

Group comparison of midterm exam scores between students who reported LLM use in prior assignments and those who did not, in a course of 68 students. The article reports the No LLM group "performing better with a large effect size" on Midterm 1; Table 2 prints d = -0.65, t = -2.5, p = .016 for that comparison.

"Overall, we found a statistically significant difference in performance on the first midterm between students who reported using LLMs on the assignments preceding this exam and students who reported no LLM use, with the latter group performing better with a large effect size."

Minju Park 2026 (2)¶

Minju Park, Ivan Orozco Vasquez, and Cristina Conati. (2026). Characterizing Students' LLM Usage Behaviors and Their Association with Learning in Critical Thinking Tasks. https://arxiv.org/abs/2605.04534

q2 · i? · associational · r1

Follow-up comparison across the three midterms in the same course dataset. The gap between LLM users and non-users "decreased for the subsequent two midterms", and the LLM-using group improved by approximately 10% from the first to the last midterm; Table 2 prints non-significant differences for Midterms 2 and 3 (p = 0.133 and p = 0.499).

"This dif- ference decreased for the subsequent two midterms and, no- tably, the LLM-using group improved by approximately 10% from the first midterm to the last."

Discussion¶

Related Claims¶

• Students received significantly higher homework scores during GenAI-evaluation periods, but this advantage did not transfer to downstream exams — the general claim this finding bears on, with the studies behind it
• Midterm self-efficacy (comfort and confidence) did not differ between conditions after three assignments, though both varied significantly across algorithmic topics — related
• Learning-by-teaching an LLM improved assignment and project performance versus historical cohorts, but not exam performance, in an undergraduate CS course — related
• Students whose LLM use was mostly student-driven support scored higher on Midterm 1 than LLM-driven users (about 10%), but the two groups converged on later midterms — related
• Among LLM users, Low-Reliance students (use in under half of submissions) consistently outperformed High-Reliance students by roughly 7-8% across all three midterms — related
• LLM usage varied across assignments and tasks, increasing for later assignments (especially discussion points), while Low-Reliance students concentrated use early and reduced it after Midterm 2 — related
• Individual trajectories suggest heavy LLM use focused on support other than full answer generation can coexist with improving scores, whereas reliance on LLM-generated summaries was associated with declining scores — related
• Flagged Spring 2026 students scored lower on all three exams than non-flagged peers, with performance declining as flagged assignments increased — related
• High-frequency LLM users show higher minimum course outcomes and reduced score variability, though interaction level is not a dominant outcome predictor — related
• Across three physics assessments, students who declared GenAI use scored slightly higher than non-users, but none of the differences were statistically significant — reports the opposite