Warning labels alone did not improve performance or reduce reliance on GenAI advice, and students sometimes adopted incorrect AI answers¶

Claim · All claims
Evidence · 1 study (2 entries) · review r2–r3 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Warnings that ChatGPT can make mistakes did not materially improve performance or reduce reliance. → Zhao J 2026 q2 i? Students sometimes changed answers to match incorrect AI advice, and peer discussion did not always repair misinformation. → Zhao J 2026 (2)

Evidence¶

Zhao J 2026¶

Zhao J, Yang K, Gao H and Jiang G (2026). Generative artificial intelligence and the transformation of medical education: a scoping review. Front. Educ. 11:1907589. https://doi.org/10.3389/feduc.2026.1907589

q2 · i? · review · r3

The review reports this behavioral-trial finding in its AI literacy and calibrated reliance section, using it to argue safeguards must be built into the learning task.

"Warnings that “ChatGPT can make mistakes” did not materially improve performance or reduce reliance (Kıyak et al., 2026a)."

Zhao J 2026 (2)¶

Zhao J, Yang K, Gao H and Jiang G (2026). Generative artificial intelligence and the transformation of medical education: a scoping review. Front. Educ. 11:1907589. https://doi.org/10.3389/feduc.2026.1907589

q2 · i? · review · r2

The review reports these reliance findings as consistent with automation-bias research, where decision support improves average performance while creating new error pathways under over-trust.

"Students sometimes changed answers to match incorrect AI advice, and peer discussion did not always repair misinformation (Kirpalani et al., 2025; Kirpalani and Wang, 2026)."

Discussion¶

Related Claims¶

• The AI-fallibility warning did not significantly change error rate or time spent per problem-solving step — possibly the same claim (merge candidate)
• AI assistance can inflate confidence even after errors and reduce accuracy when AI is faulty (cognitive surrender) — a broader claim this one bears on
• Verification bottleneck: as reliance on AI increases, accuracy in detecting AI errors decreases while confidence does not — related
• A warning message about potential AI mistakes increases seventh-graders' hint requests in a math intelligent tutoring system — related
• Students report negative experiences with ChatGPT including inaccurate outputs, lack of emotional connection, and risk of overreliance, which constrain competence — related
• Learner-level challenges of ChatGPT in STEAM IBL include over-reliance, inaccurate or irrelevant feedback, conceptual confusion, and diminished creativity — a broader claim this one bears on
• Better awareness calibration co-occurs with lower incorrect-advice acceptance and lower attribution bias, interpreted associationally only — related
• A brief metacognitive reflection prompt reduces acceptance of incorrect AI advice compared with open ChatGPT support alone — related
• Providing LLM pre-labels can decrease human annotation accuracy when the model is incorrect, due to automation bias — related
• Warning-only LA feedback can increase mental load, whereas pairing warnings with encouragement leads to higher emotional and cognitive engagement — related