General-purpose LLMs show accuracy and reliability limits, including hallucinations and task-specific accuracy below perfect, making them risky as expert advice sources¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Investigations found ChatGPT achieved about 85% accuracy on one task and endorsed incorrect statements up to 26% of the time on another, and users interacting with LLMs performed no better than controls at identifying relevant medical conditions. → Potel 2026

Evidence¶

Potel 2026¶

Potel, J. T., & Kumashiro, M. (2026). Artificial Intelligence as a Personal Coach: A Narrative Review of Benefits and Risks in Educational and Health Contexts. Behavioral Sciences, 16(8), 1431. https://doi.org/10.3390/bs16081431

q2 · i? · review · r2

The review cites task- and model-specific accuracy investigations (Ding et al.; Khatun & Brown), and Bean et al. finding users with the same LLMs performed no better than controls despite accurate stand-alone model descriptions.

"one recent investigation found that ChatGPT only achieved about 85% accuracy, occasionally changing its answers from correct to incorrect or vice versa upon receiving additional questions"

Discussion¶

Related Claims¶

• Reviewed literature reports risks of overreliance, AI errors in complex tasks, and students' uncritical acceptance of AI-generated outputs — a broader claim this one bears on
• The most reported risks of AI tools in science and chemistry education are ethical issues, including gender and racial bias, hallucinations, copyright infringement, and plagiarism — a broader claim this one bears on
• Students report negative experiences with ChatGPT including inaccurate outputs, lack of emotional connection, and risk of overreliance, which constrain competence — related
• Zero-shot LLM errors follow four recurring patterns: over-interpretation, failure to detect relevant information, hallucination, and failure to generate a response — a narrower finding that bears on this claim
• The most frequent limitation of AI in instructional design is content reliability and transparency (16 of 28 studies, 57.14%) — a broader claim this one bears on
• Teachers report ChatGPT-generated math questions misalign with grade level and topic, are unreliable or unanswerable, and cost time — related
• LLMs in educational applications hallucinate in 15–25% of responses and can amplify bias against non-native language inputs — related
• Students perceive LLM chatbots as useful for learning but requiring critical use, reporting hallucinations, overconfidence, and dependency risks — related
• ChatGPT shows high success on open-ended general chemistry exam questions but lower accuracy at the university level, and unverified copying of its answers is a reported risk — related
• Current LLMs misalign instructional-strategy sensitivity with pedagogical relevance, including hallucinated relevance of irrelevant traits — a narrower finding that bears on this claim
• Users rate chatbots easy to use with clear, relevant, largely accurate responses, but the intrusive pop-up design is the largest annoyance — related
• Large language models can produce fluent but factually inaccurate or fabricated outputs (AI hallucination), posing risks for uncritical users in science education — a narrower finding that bears on this claim
• A physics-education review reports one study (Ding et al. 2023) in which high-trust students agreed with ChatGPT's answers 100% of the time although those answers were 82% accurate — a narrower finding that bears on this claim