AI support can improve task performance while weakening users' metacognitive accuracy and leaving their own reasoning unimproved¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r? · q1
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 2 of 2 entries) · not reviewed by a person

Subclaims¶

q1 i? LLMs improved task performance but made users overconfident about their own accuracy, and most participants relied on single prompts without further follow-up. → Sander de Jong 2026 q1 i? Giving physicians access to an LLM did not improve their diagnostic reasoning even though the LLM outperformed them. → Sander de Jong 2026 (2)

Evidence¶

Sander de Jong 2026¶

Sander de Jong. (2026). Open Questions Towards Skill-Sustaining Reliance in Reflective AI Engagement. Workshop paper presented at CHIWORK'26 Workshop on Human Agency and Skill in AI-Supported Work. https://arxiv.org/abs/2609.27726 doi:10.48550/arxiv.2609.27726

q1 · i? · theoretical · r3

A workshop position paper that poses open questions; it runs no study and has no review method. In Section 4 it reports, second-hand and citing Fernandes et al., that LLMs improved task performance but made users overconfident about their own accuracy, and that most participants relied on single prompts without follow-up, which the author reads as the interaction not encouraging reflection. The finding is Fernandes et al.'s, not this paper's.

"Fernandes et al., who found that LLMs improved task performance but made users overconfident about their own accuracy [11]. Most participants relied on single prompts without further follow-up"

Sander de Jong 2026 (2)¶

Sander de Jong. (2026). Open Questions Towards Skill-Sustaining Reliance in Reflective AI Engagement. Workshop paper presented at CHIWORK'26 Workshop on Human Agency and Skill in AI-Supported Work. https://arxiv.org/abs/2609.27726

q1 · i? · causal · r?

Narrative review attribution in Section 5: the paper reports, citing Goh et al., that physician access to an LLM did not improve their diagnostic reasoning despite the LLM outperforming them, suggesting the interaction did not translate AI capability into better human reasoning.

"Goh et al. found that giving physicians access to an LLM did not improve their diagnostic reasoning, even though the LLM outperformed them"

Discussion¶

Related Claims¶

• Reviewed literature reports risks of overreliance, AI errors in complex tasks, and students' uncritical acceptance of AI-generated outputs — a broader claim this one bears on
• Unguarded answer-giving AI harmed unaided exam performance while a guarded version of the same model erased the harm (Bastani et al., 2025, as reported) — related
• Students perceive LLM chatbots as useful for learning but requiring critical use, reporting hallucinations, overconfidence, and dependency risks — related
• The article reports observational patterns that AI assistance can improve short-term performance while reducing cognitive engagement and independent capability — a broader claim this one bears on
• Access to an LLM assistant during practice did not significantly lower subsequent unaided test performance — related
• Fluent Illusions Mislead Self Assessment — related
• A theoretical article, citing two simulation studies, reports that novice clinical learners overestimated their performance and that their confidence exceeded their accuracy — related
• Longitudinal studies of trust and reliance in AI advice examine whether users rely appropriately over time but not whether independent reasoning capacity is maintained — related
• The paper reports that metacognition—students thinking about how they understand their own learning—supports more efficient and effective learning and improved outcomes — related
• Optimizing prompts can vastly improve the accuracy of LLM outputs, but critical evaluation of the resulting outputs still requires domain expertise. — related
• AI-related placebo effects lead users to overestimate their own abilities while or after being assisted by an AI — related