A scaffolded AI Study Coach produced no assignment-performance or concept-inventory advantage over unrestricted AI use in a master's programming course pilot¶

Claim · All claims
Evidence · 1 study · causal r1 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q2 i? Students working with the scaffolded Coach and students with unrestricted AI did not differ in rubric-scored assignment performance or concept-inventory gains. → Azimi 2026

Evidence¶

Azimi 2026¶

Azimi, S. (2026). AI Literacy over Tool Design: A Mixed-Methods Study of Scaffolded versus Unrestricted Generative AI in Programming Education. Delft University of Technology. https://arxiv.org/abs/2609.16784 doi:10.48550/arxiv.2609.16784

q2 · i? · causal · r1

Descriptive comparison in a seven-week pilot with 33 master's students randomly assigned to a Coach condition (n=16) or unrestricted AI (n=17); no inferential tests were run. The article reports "Gains on the concept inventory between week 2 and week 7 were similar in both groups" and equivalent rubric scores.

"The two conditions did not differ in assignment performance. Gains on the concept inventory between week 2 and week 7 were similar in both groups, and the rubric scores on the assignments were equivalent."

Discussion¶

Related Claims¶

• Contingent scaffolding improves learning more than fixed or absent support. — the general claim this finding bears on, with the studies behind it
• Students in the Coach condition reported higher confidence in their own competence and a larger confidence gain over the seven weeks than unrestricted-AI students — related
• The Coach's hint budget was managed poorly: students split into early-spenders and savers, and within the Coach condition only those with a deliberate hint strategy scored higher — related
• Students with the best self-taught understanding of how language models work used AI tools most deliberately and achieved the highest assignment scores — related
• Students who formulated their own rules for when to use AI performed better on assignments in both conditions — related
• Students in both conditions named awareness of their reliance on AI as the most valuable course outcome, and unrestricted-AI students described shallow engagement they could not stop — related
• Unrestricted ChatGPT access yields better practice performance but significantly worse exam scores — reports the opposite
• AI-generated homework hints were positively associated with exam performance only when exam questions closely matched the homework content — related
• Falsifiable hypothesis: unrestricted AI use on deep-processing tasks will produce a product–process dissociation—higher-rated assignments but lower unaided delayed transfer — related
• Delayed final-exam differences favored Guided-LLM descriptively but were not statistically confirmatory, with confidence intervals including zero — related
• Prompt-literacy guidance was associated with higher transcript-coded rule-following compliance than unrestricted LLM access, mainly in process-over-answer and stepwise-hint behaviors — related
• Live feedback shifts the router's strategy from the Feynman prompt favored in simulation to the scaffolding Coach prompt, with superior live performance — related
• Learning-by-teaching an LLM improved assignment and project performance versus historical cohorts, but not exam performance, in an undergraduate CS course — related