Surface validation can function while substantive validation fails: most participants detected flawed AI recommendations yet continued following them¶

Claim · All claims
Evidence · 1 study · theoretical r3 · q3
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? In an experiment on biased AI recommendations, 80.7% of participants detected errors, demonstrating functional surface validation, but they continued following the erroneous advice, indicating that detecting a problem and overriding an algorithmic recommendation require different cognitive capacities. → Lovett 2026

Evidence¶

Lovett 2026¶

Lovett, N. (2026). The tragedy of the cognitive commons: How AI could disrupt the regeneration of professional expertise. Human Resource Development Review. Advance online publication. https://doi.org/10.1177/15344843261470602

q3 · i? · theoretical · r3

The article reports Vicente and Matute's (2023) experiment: participants exposed to biased AI recommendations reproduced those biases in later judgments even after assistance was removed, with "80.7% of participants detected errors" yet deferred anyway, treating the system as an authority. No effect size is printed.

"Critically, 80.7% of participants detected errors in the AI recommendations, demonstrating functional surface validation. But they continued following erroneous advice, suggesting that detecting a problem and overriding an algorithmic recommendation require different cognitive capacities"

Discussion¶

Related Claims¶

• Reviewed literature reports risks of overreliance, AI errors in complex tasks, and students' uncritical acceptance of AI-generated outputs — a broader claim this one bears on
• Verification bottleneck: as reliance on AI increases, accuracy in detecting AI errors decreases while confidence does not — related
• AI functioned primarily as a source of preliminary ideas and cognitive support rather than as a substitute for professional judgment, with students identifying contextual limitations of AI recommendations — reports the opposite
• Students' critical examination of AI-generated outputs increased across learning cycles, though unevenly, with some participants continuing to seek confirmation from AI — related
• Perfect AI execution cannot remove four limits: accuracy does not settle end-setting, safety does not confer legitimacy, detection does not secure integrity, and outputs do not replace participation — related
• Source provenance labels functioned as a seal of authority: 11 of 12 participants failed to detect a planted source-card mismatch — related
• LLM output can be misleading or erroneous because a stochastic element in text generation sometimes selects less likely continuations, so it requires continuous supervision and critical evaluation by domain experts. — related
• A first evaluation of 3B confirms its conformance to critical usability requirements including safety and comfort, believability, simplicity, acceptability, extensibility, affordability, and mobility — related
• Optimizing prompts can vastly improve the accuracy of LLM outputs, but critical evaluation of the resulting outputs still requires domain expertise. — related
• Experimental-group students detected AI errors and enacted verification and ethical self-regulation as routine practices — reports the opposite
• Students' reliance on surface-level evaluation strategies declined after the modules, including trust in dot-org domains, links, appearance, and About pages — related