Reviews citing one randomised trial (Bastani et al. 2025) report that unrestricted GPT-4 access in high-school mathematics raised assisted practice performance but left students worse off after withdrawal¶

Claim · All claims
Evidence · 4 studies · 2 theoretical r3, 1 review r2, 1 qualitative r1 · q1–q2
Currency · newest study 2026 · checked against its sources 2026-10-17 (automated, 4 of 4 entries) · not reviewed by a person

Subclaims¶

q2 i? In a randomised controlled trial with nearly a thousand high-school mathematics students, GPT-4 access raised assisted practice performance by 48 per cent, yet after access was withdrawn the same students performed 17 per cent worse than controls. → Mutlu Cukurova 2026

q2 i? The whitepaper reports that a randomized controlled trial with nearly 1,000 students found unrestricted GPT-4 access improved practice performance by 48% but led to a 17% decline in exam scores when AI was removed, and that a guardrail version providing hints rather than direct answers mitigated these negative effects. → Reshaping Undergraduate Computer Science Education in the Generative AI Era 2026

q1 i? The review reports, citing Bastani et al. (2025), that relative to traditional instruction, generative AI raised students' immediate performance but impaired delayed exam scores when AI access was removed. → Zemin Guo 2026

q2 i? Students given access to an AI assistant in high school mathematics showed improved problem-solving during learning but significant harm to durable independent learning once the AI was removed. → Khosravi 2026

q2 i? In a large randomised high school mathematics experiment, students with AI access improved problem-solving during learning but showed significant harm to durable independent learning after AI removal. → Khosravi 2026

Evidence¶

Mutlu Cukurova 2026¶

Mutlu Cukurova. (2026). What do you mean by 'human–AI collaboration'? Prerequisite functions and the affordances needed to achieve it. Submitted to the Handbook of AI and the Future of Education. https://orcid.org/0000-0001-5843-4854

q2 · i? · theoretical · r3

The chapter reports, citing Bastani et al., a randomised controlled trial with nearly a thousand high-school mathematics students, where "unrestricted access to a GPT-4 interface raised assisted practice performance by 48 per cent" while withdrawal left students 17 per cent worse off than never-users.

"unrestricted access to a GPT-4 interface raised assisted practice performance by 48 per cent; yet when access was subsequently withdrawn, the same students performed 17 per cent worse than controls who had never used the tool"

Reshaping Undergraduate Computer Science Education in the Generative AI Era 2026¶

Reshaping Undergraduate Computer Science Education in the Generative AI Era. (2026). https://arxiv.org/abs/2606.07545

q2 · i? · qualitative · r1

The whitepaper reports this second-hand, citing Bastani et al.: a randomized controlled trial with nearly 1,000 students in which "unrestricted GPT -4 access" raised practice performance 48% but exam scores fell 17% once AI was removed. The whitepaper does not name the students' level or subject in this passage; its reference list gives the trial's title as "Generative AI without guardrails can harm learning: Evidence from high school mathematics". The 48% and 17% are percentage differences, not a d-family effect size, so impact is coded i? (not reported), which is not a null effect.

"Bastani et al. [3] conducted a randomized controlled trial with nearly 1,000 students and found that unrestricted GPT -4 access, while improving practice performance by 48%, led to a 17% decline in exam scores when AI was subsequently removed."

Zemin Guo 2026¶

Zemin Guo, Wenhui Li, Laure Lu Chen, Qianru Liang, & Yuxuan Wu. (2026). Reconceptualizing digital literacy in the AI era: A systematic review of validated frameworks from an ecological perspective. Computers and Education: Artificial Intelligence, 11, 100679. https://doi.org/10.1016/j.caeai.2026.100679

q1 · i? · review · r2

The review attributes this finding to Bastani et al. (2025) in its micro-level ecology section, using it to illustrate the performance–mastery distinction: AI may inflate observable performance "without necessarily producing durable acquisition." In its text the review names neither the model (GPT-4) nor an unrestricted-access condition, and does not describe the trial's design; it says only "generative AI". The subject and level appear only in its reference list, which titles the study Bastani et al. (2025), "Generative AI without guardrails can harm learning: Evidence from high school mathematics". The underlying study was not read for this extraction, so GPT-4, unrestricted access and the randomised design are not established by this entry.

"Bastani et al. (2025) found that, relative to traditional instruction, generative AI could raise students' immediate performance but impair delayed exam scores when AI access is removed."

Khosravi 2026¶

Khosravi, H., Gašević, D., Sadiq, S., Yan, L., Lodge, J. M., Tangen, J. M., Denny, P., DiCerbo, K., Buckingham Shum, S., & Baker, R. S. (2026). Building AI Companions that Prioritise Learning over Performance. arXiv. https://arxiv.org/abs/2605.04816. doi:10.48550/arxiv.2605.04816

q2 · i? · theoretical · r3

The paper reports, citing a large randomised experiment in high school mathematics (its reference [10], Bastani et al., 2025), that AI access improved in-task problem solving but harmed durable independent learning after removal. It calls the tool only "an AI assistant": it does not name GPT-4 or say access was unrestricted. No effect size is printed in the supplied text.

"students given access to an AI assistant showed improved problem-solving during learning but suffered significant harm to their durable, independent learning once the AI was removed"

Does performing with an aid at hand leave learners able to do the task once the aid is removed? This table gathers the studies behind this claim and the 8 narrower claims that bear on it: 11 distinct studies (5 causal, 2 theoretical, 2 review, 1 qualitative, 1 design), q1–q4, 1 of 11 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Every entry this page had was a second-hand report of one trial (Bastani et al., 2025, high-school mathematics, nearly 1,000 students), so they are kept as one entry; the other five reports are in comments under it. The trial itself has no entry here yet.

Related Claims¶

• Observable output alone is insufficient evidence of learning when generative AI can produce sophisticated work without underlying cognitive change — a broader claim this one bears on
• Review reports a randomized trial in which unrestricted ChatGPT study produced lower delayed retention than traditional methods — related
• AI-induced gains are not purely delegation: treated participants do not perform worse once AI is removed, and lower-education participants retain part of their gain, though a 0.200 SD gap re-emerges — reports the opposite
• AI access raises immediate unaided test scores by 7.2 percentage points (0.28 SD) after a timed learning phase — related
• In a randomized experiment on fraction problems, participants who used an AI assistant solved fewer unassisted test problems and skipped more than control participants — related
• Among GPT-assisted students, proactive and critical engagement patterns are associated with higher exam scores while limited engagement shows little benefit relative to control — related
• High-school students given GPT-4 access solved more problems in the moment but performed worse than controls once the tool was removed (review attribution) — possibly the same claim (merge candidate)
• Access to a standard GPT-4 chat tutor during practice substantially improves practice performance in high school mathematics — related
• A guardrailed GPT-4 tutor with teacher-designed hints improves assisted practice performance more than a standard GPT-4 interface — related
• Students do not perceive the learning harm from copying GPT solutions and overestimate their exam performance — related
• Students given AI assistance in peer feedback tended to rely on the AI-generated feedback and struggled to perform the task when the assistance was removed — related
• Falsifiable hypothesis: unrestricted AI use on deep-processing tasks will produce a product–process dissociation—higher-rated assignments but lower unaided delayed transfer — a broader claim this one bears on
• Performance improvements from generative AI assistance diminished once the assistance was removed — a narrower finding that bears on this claim
• LLMs perform worse on complex, open-ended, or subjective assignments than on short, clearly formulated answers — related
• A scaffolded AI Study Coach produced no assignment-performance or concept-inventory advantage over unrestricted AI use in a master's programming course pilot — reports the opposite
• ChatGPT's exam scores were within 10% of human grades 70% of the time in a study of AI-based grading — related
• Random assignment to AI access produces a strong first stage: about 70 percent of treated students use AI, most often to explain concepts — related
• Dependent versus autonomous cognitive offloading to GenAI shows opposite motivational and cognitive outcome patterns — related
• Students received significantly higher homework scores during GenAI-evaluation periods, but this advantage did not transfer to downstream exams — related
• AI-generated homework hints were positively associated with exam performance only when exam questions closely matched the homework content — related
• A carefully engineered AI tutor produced more than twice the learning of a strong active-learning class, in less time (Kestin et al., 2025, as reported) — related
• AI assistance that substitutes for learning activities may improve immediate performance while impeding long-term skill development — a broader claim this one bears on
• AI assistance reduces persistence: persistence costs concentrate among learners who used AI for direct solutions, not hints — related
• In one RCT, a content-rich prompt-engineered generative AI chat tutor improved performance and engagement in a Harvard physics course — related
• One week later, the AI-access group's lead on unaided assessments shrinks to 3.9 pp (0.21 SD), short of statistical significance (p = 0.068) — reports the opposite
• Hint-not-solution is the most challenging rubric dimension, with scores ranging from 6% to 81% across models — related
• For exam-oriented learners, trust in AI tutors hinges on curriculum fit: students abandon mathematically correct solutions that use out-of-syllabus methods — related
• Prompt-literacy guidance was associated with higher transcript-coded rule-following compliance than unrestricted LLM access, mainly in process-over-answer and stepwise-hint behaviors — related
• Students learn more with a research-based AI tutor than with in-class active learning on identical content (post-test scores) — related
• LLM-assisted self-study yields higher post-intervention exam performance than traditional resources among early-career knowledge workers — reports the opposite
• Persistence and performance costs concentrate among participants who used the AI to obtain direct solutions, while hint-users resemble controls — related
• Greater answer offloading during learning is associated with lower subsequent unaided performance (exploratory) — related
• Outsourcing productive struggle to AI risks cognitive debt, and reactive compliance-focused approaches are insufficient — a broader claim this one bears on
• Access to an LLM assistant during practice did not significantly lower subsequent unaided test performance — reports the opposite
• In one preregistered experiment, metacognitive feedback on learners' own LLM use lowered the odds of answer offloading and raised the odds of a correct unaided answer on one-sided tests, with 95% intervals reaching no effect — related
• In math problem solving, a tutor-plus-peers LLM arrangement produced the strongest unassisted post-task test accuracy, with accuracy increasing monotonically from Control to Tutor + Peers — related
• How a user engages with AI matters more than whether: structured, deliberate engagement can mitigate or reverse offloading effects — a broader claim this one bears on
• Most introductory physics students preferred a combination chatbot (guidance first, answers when stuck), while few preferred pure guided Q&A — related
• AI support can improve task performance while weakening users' metacognitive accuracy and leaving their own reasoning unimproved — related
• Constituting AI as a critical file that returns questions instead of revisions is a design requirement for sustaining productive struggle — a narrower finding that bears on this claim
• Free ChatGPT use boosts originality on the assisted creative task but the advantage disappears when assistance is removed — related
• Immediate task success does not establish learning that persists without AI support — related
• Teachers sought to preserve student thinking and authorship by bounding chatbot answer-giving and maintaining teacher control over scope — related
• Performance during the AI-supported task consistently predicted performance on the subsequent AI-removal task across all writing dimensions — reports the opposite
• GenAI/LLM modalities provide the strongest evidence for cognitive and communication outcomes, while chatbots show weak independent effects — related
• Students use standard GPT-4 access as a crutch—copying answers rather than engaging with the material—which explains the harm to learning — related
• Intensive LLM use in educational contexts may lead to an accumulation of cognitive debt that weakens critical thinking and epistemic ownership — related
• Positive LLM tutor perceptions do not necessarily translate to learning gains, creating an over-reliance risk — related
• Prior genAI access shows only exploratory evidence of poorer subsequent unaided reasoning, within the no-incentive condition, and no preregistered main effect — a narrower finding that bears on this claim
• Easy access to AI-generated answers may reduce learners' motivation to engage deeply with material — a broader claim this one bears on
• Meta-reviews reach contradictory conclusions on whether AI reliance harms or helps critical thinking, with substantial design limitations limiting generalizability — related
• Agentivism generates six empirically testable propositions linking delegation structure, verification, reconstruction, and reduced-support transfer to learning outcomes — a broader claim this one bears on
• Calibrated epistemic vigilance, given adequate prior knowledge, is the binding constraint on productive augmentation with AI — related
• Productive cognitive offloading to AI can shift into cognitive surrender, abdicating learner intellectual agency — a broader claim this one bears on
• Scaffolding improves learning — a broader claim this one bears on
• Minimal guidance is less effective for novices than explicit instruction — a broader claim this one bears on
• In one experiment with 256 undergraduates, computer-based lesson versions that included practice produced significantly higher posttest scores than versions without practice — related
• Review reports a developmental shift in cognitive offloading: children under-use external aids while adolescents over-rely on them — related
• Practicing with AI improved unassisted writing skill more than practicing with Google Search for cover letter examples and tips — related
• Tutors with Tutor CoPilot access use more understanding-fostering strategies (e.g., prompting students to explain) and rely less on giving away answers than control tutors — a narrower finding that bears on this claim
• Engagement remains an observable and evidential proxy for learning that AI has not broken, though it is not invulnerable — related
• Expected tangible rewards reduce later free-choice engagement with a rewarded task, while verbal praise does not — a broader claim this one bears on
• In TBLT, teachers perceive scaffolding collapse, a shortcut paradox, invisible inequity, and task realism corrosion — a narrower finding that bears on this claim
• A systematic review of 14 studies (8,305 participants) found AI-based tools enhanced comprehension, engagement, problem-solving, and critical thinking by personalizing learning experiences — related
• A systematic review of AI in problem-based learning reports, as a dominant theme of its 50 studies, that AI tools support adaptive, personalized learning linked to better engagement and problem-solving — reports the opposite
• AI task scaffolding is positively associated with critical thinking and technical problem-solving among AI-using university students — related
• Frequent generative AI use enhances perceived confidence and efficiency but increases technological dependence among university students — related
• High-achieving students less often endorse AI-supported autonomy, effective learning, and active engagement, and more often agree that reliance on AI hinders critical thinking and independent problem solving — related
• Effectiveness of AI-driven personalized learning is context-dependent, with CS and Mathematics students showing contrasting engagement and performance trajectories — related
• The article reports observational patterns that AI assistance can improve short-term performance while reducing cognitive engagement and independent capability — a broader claim this one bears on
• Substituting tasks with generative AI may promote inclusion and free time for more demanding activities, but risks foundational-skill loss and presupposes prior knowledge — a narrower finding that bears on this claim
• More teacher support in AI-based instruction is not always better; a review found support worked well when balanced with students' autonomy and independent thinking — related
• In simulation, AI-assisted students showed increased scores and reduced solving time, attributed to personalized assistance — reports the opposite
• GenAI literacy predicted immediate independent writing performance after AI removal for Visual Data Integration, Critical Thinking, and overall Composite Score — related
• Staff observed increased student engagement, independence, and communication skills during the AI-supported project — related
• Personalization reduced time spent on hard expression-writing steps, and students who had received it were still faster at writing expressions in a later non-personalized unit — related
• Undergraduates learning IoT tasks in VR with ChatGPT-integrated feedback reached significantly higher cognitive levels on memory, understanding, and application tasks than peers receiving traditional VR feedback — related
• AI assistance impairs unassisted performance and persistence in reading comprehension, replicating the fraction-task effects in a second domain — related
• Developers who used AI assistance scored 17% lower on a subsequent comprehension assessment than developers who completed the same tasks without AI, per the cited controlled study — possibly the same claim (merge candidate)
• AI-assisted performance during skill-building does not transfer to subsequent unassisted performance — related
• The review reports AI-driven personalization improves student engagement and learning outcomes, and virtual teaching assistants support diverse learning needs — reports the opposite
• Practicing cover letter writing with an AI tool improved unassisted writing skill more than practicing without AI or not practicing, with benefits persisting one day later — related
• AI raises the ceiling on problem complexity by reducing access complexity rather than cognitive demand — related
• The whitepaper reports, citing He et al., that Cursor adoption was associated with a persistent increase in static analysis warnings and code complexity despite short-term velocity gains — related
• The paper reports that metacognition—students thinking about how they understand their own learning—supports more efficient and effective learning and improved outcomes — related
• Guardrails in the GPT Tutor largely eliminate the negative exam effect, though no positive exam effect is observed — related
• Descriptive user-study patterns that the authors read as GPT output possibly scaffolding IVDA students and reducing CV students' performance, an interpretation by prior knowledge that the study did not measure — related
• Feedback source (human tutor vs GPT-4) had no significant effect on ENL students' posttest writing scores — related
• Unfettered GPT-4 access during practice harms subsequent unassisted exam performance in high school mathematics — possibly the same claim (merge candidate)
• Students receiving instructor feedback showed significantly greater improvements in lab assignment scores than students receiving AI-generated feedback, after controlling for initial knowledge — related
• The regulated 'think first, ChatGPT later' group outperformed both other groups in independent originality on the unassisted task despite no assisted-task advantage — related
• The tutor-access learning gain was concentrated in conceptual depth of written answers rather than multiple-choice scores — related
• Aiding-based transfer differs by trainee experience: first-semester trainees showed no aided-to-unaided transfer on Task One and initial negative then positive transfer on Task Two — related
• AI-generated (GPT-4) feedback produced no greater linguistic progress than human tutor feedback among ENL students over six weeks — related
• A quarter of students offered a free AI tutor never tried it once, making first-session adoption a deployment problem — related
• A general-purpose LLM used as a tutor without pedagogical prompting overwhelmingly provides high assistance, with more than 95% of responses classified as Explaining or Solving — related
• Ambient consumer AI use in the holdout did not close the structured-tutor gap — related
• In math problem solving, a tutor LLM alone significantly improved unassisted test accuracy over no-agent control, while peer-like error agents alone did not reach significance — related
• Chatbot access during mathematics practice improved practice performance but lowered subsequent unaided exam performance, and a hint-giving interface eliminated the penalty — possibly the same claim (merge candidate)
• AI over-reliance can produce inversion effects, reducing cognitive processing depth and learning outcomes — a narrower finding that bears on this claim