AI assistance can inflate confidence even after errors and reduce accuracy when AI is faulty (cognitive surrender)¶

Claim · All claims
Evidence · 1 study · theoretical r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q2 i? Experimental evidence cited by the paper shows AI assistance can increase accuracy when AI is correct, reduce accuracy when AI is faulty, and inflate confidence even after errors. → Dongming Mei 2026

Evidence¶

Dongming Mei 2026¶

Dongming Mei, Katherine Moore, and Ben Sayler. (2026). Preparing Students for AI-Powered Materials Discovery: A Workflow-Aligned Framework for AI Literacy, Equity, and Scientific Judgment. https://arxiv.org/abs/2605.09624 doi:10.48550/arxiv.2605.09624

q2 · i? · theoretical · r2

The paper reports, citing Shaw and Nave's experimental work, that 'AI assistance can increase accuracy when AI is correct, reduce accuracy when AI is faulty, and inflate confidence even after errors.' No effect sizes are printed in the paper's account.

"Their experimental evidence shows that AI assistance can increase accuracy when AI is correct, reduce accuracy when AI is faulty, and inflate confidence even after errors"

Discussion¶

Related Claims¶

• Access to AI advice roughly doubled reported confidence in answers in one experiment — related
• Monetary incentives for accuracy improve answer correctness specifically when AI advice is available, by reducing reliance on that advice — related
• Monetary stakes for accuracy act independently of AI availability on judgment suspension, with no significant AI × stakes interaction — related
• Participants seek AI advice less frequently when accuracy carries monetary stakes — related
• Merely having access to AI advice nearly eliminates people's willingness to suspend judgment under uncertainty, even when the advice is wrong — related
• AI advice dramatically reduces judgment suspension even when it is displayed automatically rather than actively requested — related
• Verification bottleneck: as reliance on AI increases, accuracy in detecting AI errors decreases while confidence does not — related
• Practitioners who adopt AI assistance show reduced independent performance compared to their baseline — related
• Warning labels alone did not improve performance or reduce reliance on GenAI advice, and students sometimes adopted incorrect AI answers — a narrower finding that bears on this claim
• Reflection produced selective rather than blanket rejection of AI: alignment with correct advice stayed high while alignment with incorrect advice dropped — a narrower finding that bears on this claim
• Post-error behavioural adjustments can occur automatically, dissociated from explicit error detection — related