Students did not consistently detect scientific errors in ChatGPT responses, and students with stronger prior knowledge evaluated AI responses more critically¶

Claim · All claims
Evidence · 1 study · review r3 · q2 · i3 large
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i3 Dahlkemper et al. (2023) found error detection was inconsistent, especially for linguistically convincing responses, and stronger prior knowledge predicted better discrimination (η2 = 0.26–0.57). → Malikova Z 2026

Evidence¶

Malikova Z 2026¶

Malikova Z, Ualikhanova B, Turmambekov T, Rakhashev B and Berdaliyev D (2026). Artificial intelligence in university physics education: a systematic review of empirical studies. Front. Educ. 11:1964629. https://doi.org/10.3389/feduc.2026.1964629

q2 · i3 · review · r3

The review reports this quantitative descriptive study of 102 university physics students in Section 3.4; Table 5 lists the reported effect size "η2 = 0.26–0.57" for knowledge-related differences in critical evaluation.

"Dahlkemper et al. (2023) found that students did not consistently detect scientific errors in ChatGPT-generated responses, particularly when the responses were linguistically convincing. Students with stronger prior knowledge were more successful in distinguishing problematic AI-generated responses from correct human-prepared answers."

Discussion¶

Learner Variables¶

• Prior Knowledge — predictor: learners who differ on it differ in outcomes

Related Claims¶

• Prior Knowledge Needed For Accurate Self Assessment — the general claim this finding bears on, with the studies behind it
• ChatGPT-generated HDR problems measure the same ability as human-created problems, with high internal consistency (Cronbach's α = 0.78) and no significant difficulty difference — related
• Verification bottleneck: as reliance on AI increases, accuracy in detecting AI errors decreases while confidence does not — related
• A physics-education review reports one study (Ding et al. 2023) in which high-trust students agreed with ChatGPT's answers 100% of the time although those answers were 82% accurate — related
• Three multilingual international students critically evaluated AI feedback, recognized peer-AI agreement, and maintained their voices while revising — related
• Students from higher-ranked universities and with stronger prior knowledge are more likely to adopt proactive and critical AI engagement — related
• Agentivism generates six empirically testable propositions linking delegation structure, verification, reconstruction, and reduced-support transfer to learning outcomes — related
• Common health literacy measures largely detect individual differences in cognitive abilities — related
• Oral assessments are recommended as a strategy to maintain academic integrity against AI use — related
• Judge-condition verification did not consistently improve LLM feedback quality and often propagated Peer agents' errors — related
• Disciplinary expertise alone is insufficient for teachers to detect GenAI representational bias — related