Developers who used AI assistance scored 17% lower on a subsequent comprehension assessment than developers who completed the same tasks without AI, per the cited controlled study¶
Claim · All claims
Evidence · 1 study · designr1·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? In a controlled study cited by the authors, developers who used AI assistance scored 17% lower on a subsequent comprehension assessment than those who completed the same tasks without AI, and full delegators showed the steepest decline in skill formation. → Rohit Mehra 2026
Evidence¶
Rohit Mehra 2026¶
Rohit Mehra, Samdyuti Suri, Prithviraj K Tagadinamani, Kapil Singi, Vikrant Kaulgud, Adam P. Burden. (2026). Agents That Teach: Towards Designing Incidental Learning Back into AI-Assisted Software Development. https://arxiv.org/abs/2607.06101 doi:10.48550/arxiv.2607.06101
q2 · i? · design · r1
The article reports, citing Shen and Tamkin (2026), a controlled study in which "developers who used AI assistance scored 17% lower on a subsequent comprehension assessment" than those working without AI. No effect size is printed; the 17% figure is the cited study's reported difference.
"In a controlled study, developers who used AI assistance scored 17% lower on a subsequent comprehension assessment than those who completed the same tasks without AI, and developers who fully delegated coding to the AI showed the steepest decline in skill formation"
Discussion¶
Related Claims¶
- Unrestricted GPT-4 access in high-school mathematics raised assisted practice performance but left students worse off after withdrawal — the general claim this finding bears on, with the studies behind it
- Students given AI assistance in peer feedback tended to rely on the AI-generated feedback and struggled to perform the task when the assistance was removed — related
- Persistence and performance costs concentrate among participants who used the AI to obtain direct solutions, while hint-users resemble controls — a broader claim this one bears on
- Immediate task success does not establish learning that persists without AI support — a broader claim this one bears on
- AI assistance reduces persistence: persistence costs concentrate among learners who used AI for direct solutions, not hints — a broader claim this one bears on
- Unfettered GPT-4 access during practice harms subsequent unassisted exam performance in high school mathematics — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
ai-assistance-lower-comprehension-17-percent |
title |
Developers who used AI assistance scored 17% lower on a subsequent comprehension assessment than developers who completed the same tasks without AI, per the cited controlled study |
description |
Developers who used AI assistance scored 17% lower on a subsequent comprehension assessment than developers who completed the same tasks without AI, per the cited controlled study |
status |
review |
evidence_strength |
weak |
newest_evidence |
2026 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — rohit-mehra-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.