Large language models can produce fluent but factually inaccurate or fabricated outputs (AI hallucination), posing risks for uncritical users in science education¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2025 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i? LLMs such as ChatGPT frequently lack contextual understanding and can generate hallucinations: grammatically sound, persuasive outputs that are factually inaccurate or fabricated. → Papaneophytou 2025

Evidence¶

Papaneophytou 2025¶

Papaneophytou, C.; Nicolaou, S.A. (2025). Promoting Critical Thinking in Biological Sciences in the Era of Artificial Intelligence: The Role of Higher Education. Trends High. Educ. 2025, 4, 24. https://doi.org/10.3390/higheredu4020024

q2 · i? · review · r2

Narrative review discussion of chatbots and LLMs in education, attributing the hallucination problem to cited work [47]. The review notes students relying uncritically could propagate errors into assessments, lab reports, or research articles.

"One critical flaw of LLMs is their tendency to produce hallucinations (this phenomenon is known as AI hallucination), outputs that are grammatically sound and persuasive but factually inaccurate or entirely fabricated [47]."

Discussion¶

Related Claims¶

• GenAI feedback limitations require students to exercise evaluative judgement
• ChatGPT-generated text shows inaccuracy, weak logical flow, lack of critical elaboration and non-originality in an author's first-hand test — a narrower finding that bears on this claim
• AI systems can produce biased or inaccurate diagnostic outputs, requiring critical teacher evaluation before integration — a narrower finding that bears on this claim
• GenAI improves administrative coordination of group work but its analyses of team members can be inaccurate and hallucinated — a narrower finding that bears on this claim
• GenAI chemistry outputs can be fluent and locally persuasive while chemically invalid across representations — a narrower finding that bears on this claim
• Students report negative experiences with ChatGPT including inaccurate outputs, lack of emotional connection, and risk of overreliance, which constrain competence — a narrower finding that bears on this claim
• The most frequent limitation of AI in instructional design is content reliability and transparency (16 of 28 studies, 57.14%) — a broader claim this one bears on
• Gen-AI hallucinates sources and values and produces convincing but incorrect explanations, risking student misinformation — a narrower finding that bears on this claim
• Students perceive LLM chatbots as useful for learning but requiring critical use, reporting hallucinations, overconfidence, and dependency risks — related
• General-purpose LLMs show accuracy and reliability limits, including hallucinations and task-specific accuracy below perfect, making them risky as expert advice sources — a broader claim this one bears on
• AI-assisted scientific writing assignments risk fabricated references and altered chemical meaning — a narrower finding that bears on this claim