Experiment 2 replicates the unassisted-performance impairment with a pretest-based exclusion, but the aggregate skip-rate difference does not reach significance¶
Claim · All claims
Evidence · 1 study · causalr2·q3·i0negligible
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i0 With pretest-based exclusions and a matched control sidebar, the AI condition still showed a lower unassisted solve rate (Cohen's d = -0.19). → Grace Liu 2026
Evidence¶
Grace Liu 2026¶
Grace Liu, Brian Christian, Tsvetomira Dumbalska, Michiel A. Bakker, Rachit Dubey. (2026). AI Assistance Reduces Persistence and Hurts Independent Performance. Published as a conference paper at COLM 2026. https://arxiv.org/abs/2604.04721 doi:10.48550/arxiv.2604.04721
q3 · i0 · causal · r2
Replication RCT (667 recruited, 585 analyzed) adding a pretest for exclusions and a control sidebar with pretest solutions to remove interface asymmetry. Solve-rate impairment replicated, "Cohen's d=− 0.19"; exclusion rates were similar across conditions (11.7% vs 12.9%).
"Participants in the AI condition had a lower solve rate (mean 0.71, s.d. 0.36) than participants in the control condition (mean 0.77, s.d. 0.32; t(583) =− 2.33, P= 0.020; Cohen's d=− 0.19, 95% CI [−0.36,−0.03])."
Discussion¶
Related Claims¶
- AI assistance impairs unassisted performance and persistence in reading comprehension, replicating the fraction-task effects in a second domain — related
- In a randomized experiment on fraction problems, participants who used an AI assistant solved fewer unassisted test problems and skipped more than control participants — related
- Persistence and performance costs concentrate among participants who used the AI to obtain direct solutions, while hint-users resemble controls — related
- AI assistance reduces persistence: persistence costs concentrate among learners who used AI for direct solutions, not hints — a broader claim this one bears on
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
ai-assistance-impairs-performance-replication-exp2 |
title |
Experiment 2 replicates the unassisted-performance impairment with a pretest-based exclusion, but the aggregate skip-rate difference does not reach significance |
description |
Experiment 2 replicates the unassisted-performance impairment with a pretest-based exclusion, but the aggregate skip-rate difference does not reach significance |
status |
review |
evidence_strength |
mixed |
newest_evidence |
2026 |
generated |
by process:wiki-ingest on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — grace-liu-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.