Reviewed literature reports risks of overreliance, AI errors in complex tasks, and students' uncritical acceptance of AI-generated outputs¶

Claim · All claims
Evidence · 1 study (2 entries) · quant-synthesis r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 2 of 2 entries) · not reviewed by a person

Subclaims¶

q2 i? Heavy dependence on ready-made AI solutions may reduce opportunities to practise problem-solving, and AI tools including specialised Math-LLMs can produce incorrect solutions or flawed strategies on complex problems. → Omirzakova 2026 q2 i? Learners often pay little attention to verifying the accuracy of AI-provided solutions, allowing misconceptions to take root; the review concludes educator oversight remains essential. → Omirzakova 2026 (2)

Evidence¶

Omirzakova 2026¶

Omirzakova, F. N., & Tileubay, S. (2026). Artificial intelligence in mathematics education: A PRISMA-based systematic literature review (2021-2025). European Journal of STEM Education, 11(1), Article 43. https://doi.org/10.20897/ejsteme/19221

q2 · i? · review · r3

Thematic synthesis of limitations for RQ2. The review adds that even specialised systems such as Math-LLMs can generate inaccuracies when constructing long chains of logic [S14], raising doubts about reliability in high-accuracy tasks.

"Heavy dependence on ready -made AI solutions may reduce opportunities for learners to practise problem -solving and critical evaluation [S15]."

Omirzakova 2026 (2)¶

Omirzakova, F. N., & Tileubay, S. (2026). Artificial intelligence in mathematics education: A PRISMA-based systematic literature review (2021-2025). European Journal of STEM Education, 11(1), Article 43. https://doi.org/10.20897/ejsteme/19221

q2 · i? · quant-synthesis · r2

Review synthesis on uncritical trust in AI outputs. The authors conclude that "uncritical trust in AI -generated information is unjustified" and that active oversight by educators remains essential.

"Evidence shows that learners often pay little attention to verifying the accuracy of AI -provided solutions [S15], allowing misconceptions to take root in the learning process."

Discussion¶

Related Claims¶

• The review found AI was effective for improving basic problem-solving but required teacher mediation to handle more complex mathematical tasks — related
• A systematic review of 14 studies (8,305 participants) found AI-based tools enhanced comprehension, engagement, problem-solving, and critical thinking by personalizing learning experiences — related
• Overreliance on AI is the most prevalent pedagogical limitation (27 studies, 50%), followed by reduced critical thinking (25 studies, 46%) — a narrower finding that bears on this claim
• Unreliable AI-generated output is the most frequently reported technical limitation of AI-supported undergraduate EFL instruction, appearing in 22 of 54 studies (41%) — a narrower finding that bears on this claim
• The most frequent co-occurrence pattern pairs AI system performance limitations with learner cognitive and autonomy risks, reported together in 25 studies (46%), as association rather than causation — related
• State AI guidance describes risks but risk-management work is at an early stage, with some documents possibly too upbeat given school readiness — related
• Students given AI assistance in peer feedback tended to rely on the AI-generated feedback and struggled to perform the task when the assistance was removed — related
• LLM-assisted inductive qualitative coding carries risks of superficial themes, broad or redundant codes, and hallucinated interpretations, so LLMs should augment rather than replace human researchers — related
• Overreliance on AWE may lower learners' critical thinking and creativity — related
• Delegating the terminal evaluation to AI is self-defeating: each hand-off returns one more output someone must accept or not — related
• Students report negative experiences with ChatGPT including inaccurate outputs, lack of emotional connection, and risk of overreliance, which constrain competence — a narrower finding that bears on this claim
• LLMs expressed unjustified confidence in recommendations based on visual inputs, asserting incorrect pin-connection diagnoses with high stated certainty — related
• Generative AI-generated hints can be as effective as human-generated hints for mathematics performance, but ensuring accuracy requires validation procedures that discard a substantial share of generated hints — related
• Optional IBM Quantum hardware execution supplied extra evidence but did not remove substantial AI assistance, as ChatGPT generated the code and could interpret returned noisy data — a narrower finding that bears on this claim
• Student reflections identify speed and scaffolding as the dominant theme, with verification burden, overreliance worries, and prompt precision also prevalent — related
• Teacher and student feedback indicates LLM-based formative assessment can reduce grading burden, surface misconceptions, and build student trust without replacing teacher judgment — related
• A 2026 text-mining review chapter reports that studies of ChatGPT in programming education find benefits where it is used with structured pedagogical guidance and teacher facilitation — related
• Surface validation can function while substantive validation fails: most participants detected flawed AI recommendations yet continued following them — a narrower finding that bears on this claim
• Unstructured ChatGPT use in programming education is linked to overreliance, reduced persistence, unreliable outputs, and academic integrity risks — a narrower finding that bears on this claim
• Deterrence layers failed because seeds changed parameters rather than task structure, scaffolding exposed solution steps, and hidden grading verified output consistency rather than authorship or understanding — related
• Students hold cautious trust in the course-specific RAG system, expressing some level of trust but not treating it as an authoritative source and indicating they would verify important information — related
• Persistent challenges include feedback inaccuracy, learner overreliance, and limited transparency — related
• AI hallucination can be turned into a pedagogical resource by making verification and collaborative fact evaluation integral to AI-supported learning — related
• Practitioners hold near-universal commitments to human oversight and critical evaluation of AI outputs — related
• When AI assistance is withdrawn, student performance tends to decline, revealing a lack of internalized learning strategies — a narrower finding that bears on this claim
• GenAI use involves substantial hidden labor of prompting, verification, and correction that remains invisible in productivity evaluation — related
• Scaffolding must be balanced: too little led to confusion and overload, while too much reduced opportunities for critical thinking — related
• Students remained cautious about relying on GenAI throughout the creative process, citing imperfect outputs and perceived loss of ownership — related
• Students often adopt AI-generated solutions quickly, bypassing the effortful meaning-making that constructivist learning requires — possibly the same claim (merge candidate)
• AI systems can produce biased or inaccurate diagnostic outputs, requiring critical teacher evaluation before integration — related
• Working with imperfect AI outputs under mentor supervision was followed by students developing skepticism and independent debugging skills — a narrower finding that bears on this claim
• Teachers experienced AI as expanding design possibilities while increasing the burden of verification and professional responsibility — related
• A meta-analysis of 69 studies reports generative AI improves academic performance with g = 0.7, but the article argues this likely reflects immediate task success rather than learning — related
• Performance improvements from generative AI assistance diminished once the assistance was removed — related
• Recurring risks of GenAI in STEAM education include hallucination, bias, superficial completion strategies, and compromised assessment validity — a broader claim this one bears on
• Language becomes an operational medium in AI-supported design, with risks of narrowing exploration — related
• Providing LLM pre-labels can decrease human annotation accuracy when the model is incorrect, due to automation bias — a narrower finding that bears on this claim
• Three multilingual international students critically evaluated AI feedback, recognized peer-AI agreement, and maintained their voices while revising — related
• Thematic coding of student reflections shows peers seen as understanding context and AI feedback seen as actionable, with one quarter of reflections expressing skepticism of AI — related
• The most frequent limitation of AI in instructional design is content reliability and transparency (16 of 28 studies, 57.14%) — related
• Learner-level challenges of ChatGPT in STEAM IBL include over-reliance, inaccurate or irrelevant feedback, conceptual confusion, and diminished creativity — possibly the same claim (merge candidate)
• LEARN reconceptualises scaffolding for GAI contexts as interactive, dialogic, and epistemically regulated interrogation of AI outputs — a narrower finding that bears on this claim
• Gen-AI hallucinates sources and values and produces convincing but incorrect explanations, risking student misinformation — a narrower finding that bears on this claim
• GenAI effects on externally assessed learning are highly heterogeneous and conditional on prior knowledge, scaffolding, verification, and task design, with pooled effects unstable under publication-bias correction — related
• More experienced AI users showed a Boilerplate Blindspot: uncritically accepting 'trivial' AI-generated code in which later defects originated — a narrower finding that bears on this claim
• ChatGPT biased information access toward solution-oriented artifacts over principle-oriented information — a narrower finding that bears on this claim
• Novice AI users showed a Trust-but-Can't-Verify pattern: accepting unfamiliar constructs they could not assess, then symptom-level debugging — a narrower finding that bears on this claim
• Secondary students most frequently use generative AI for step-by-step theoretical explanation and for checking their own solutions, more than for calculations — related
• AI-assisted outputs are more similar to each other than purely human-authored ones, indicating a homogenization risk — related
• Outsourcing productive struggle to AI risks cognitive debt, and reactive compliance-focused approaches are insufficient — a narrower finding that bears on this claim
• The article claims LLM-assisted workflows weaken the validity of output-based evaluation in hiring, education, and certification — related
• Students' generative AI use may function either as a scaffold supporting thinking or as a shortcut bypassing essential cognitive processes — related
• AI use in training carries risks of uncritical acceptance and overreliance, including undetected conceptual errors and increased delegation of problem solving — possibly the same claim (merge candidate)
• ChatGPT's intuitive interface and instant feedback can either support learning or enable superficial task completion depending on integration — related
• Perfect AI execution cannot remove four limits: accuracy does not settle end-setting, safety does not confer legitimacy, detection does not secure integrity, and outputs do not replace participation — related
• Generative AI weakens the evidentiary relation between assessment artifacts and learner capacity, so institutions must assess the learner's accountable relation to AI-mediated work — related
• Before instruction, physics students show skepticism about LLM learning impacts, agreeing that over-reliance risks dependence, false confidence, and reduced conceptual understanding — a narrower finding that bears on this claim
• Problem-solving-prepared students often accepted the AI solution uncritically or critiqued it based on their own misconceptions — a narrower finding that bears on this claim
• Better awareness calibration co-occurs with lower incorrect-advice acceptance and lower attribution bias, interpreted associationally only — a narrower finding that bears on this claim
• The article argues that GAI systems' adaptive functionality inherently encourages over-reliance rather than fostering learner-centric data collection for SDL — related
• Pedagogically scaffolded generative AI in design thinking education augments rather than replaces human judgment — related
• Reflection produced selective rather than blanket rejection of AI: alignment with correct advice stayed high while alignment with incorrect advice dropped — a narrower finding that bears on this claim
• In unguided GenAI collaboration, ethnic minority preparatory students’ strategy talk co-occurred with passive sharing and uncritical acceptance, forming a plan-and-paste loop — a narrower finding that bears on this claim
• AI support can improve task performance while weakening users' metacognitive accuracy and leaving their own reasoning unimproved — a narrower finding that bears on this claim
• Students describe concrete verification behaviors for GenAI explanations, treating evaluation as an iterative filter rather than a single accept-or-reject judgment — related
• LLM-based simulated students surpass rule-based systems in natural language generation and situational flexibility, but concerns about bias and evaluation reliability persist — related
• Reflective strategies such as cognitive forcing, pre-commitment, and uncertainty communication reduce overreliance on AI and improve decision quality, but have been evaluated only as short-term single-session interventions — related
• Trained students more often identify and remediate GenAI deficiencies such as visual errors and reasoning flaws — a narrower finding that bears on this claim
• Checking can coexist with unresolved errors: 7 of 15 students failed a task containing plausible LLM falsehoods despite self-checking — a narrower finding that bears on this claim
• AI-mediated SSH learning externalizes interpretive cognition as students delegate thematic identification, contextual explanation, and comparative analysis to AI systems — a narrower finding that bears on this claim
• Students frequently accepted misleading AI recommendations (overreliance) while rarely rejecting correct ones (underreliance) — a narrower finding that bears on this claim
• Accurate LLM solution diagnosis does not reliably produce pedagogically actionable feedback, with lowest rubric scores on error identification and actionability — related
• Efficiency trap: human critical thinking may decline when AI performs most of the cognitive work — related
• GenAI improves administrative coordination of group work but its analyses of team members can be inaccurate and hallucinated — a narrower finding that bears on this claim
• An efficiency paradox: well-intentioned students default to passive GenAI processes and experience cognitive laziness and overreliance — a narrower finding that bears on this claim
• Immediate LLM explanations risk bypassing the generative processes required for durable understanding — a narrower finding that bears on this claim
• AI introduced without pedagogical scaffolding can replace rather than reinforce learners' self-regulatory processes — related
• Viewers of learning-oriented content describe ChatGPT as a tutor or thinking partner, while viewers of output-oriented content raise over-reliance and cognitive-offloading concerns — a narrower finding that bears on this claim
• AI intuition emphasizes epistemic judgment rather than output optimization, distinguishing it from prompt engineering — related
• GenAI feedback limitations require students to exercise evaluative judgement — related
• Generative AI makes knowledge mobilizable, converting external knowledge into immediately usable task output within seconds — related
• Students exhibited high trust in GenAI but limited critical evaluation of its outputs — related
• General-purpose LLMs show accuracy and reliability limits, including hallucinations and task-specific accuracy below perfect, making them risky as expert advice sources — a narrower finding that bears on this claim
• Intensive LLM use in educational contexts may lead to an accumulation of cognitive debt that weakens critical thinking and epistemic ownership — related
• Positive LLM tutor perceptions do not necessarily translate to learning gains, creating an over-reliance risk — related
• Teacher educators show shallow understanding of GenAI tools and gaps in aligning AI use with pedagogical goals — related
• Unstructured LLM use risks cognitive offloading that displaces effortful thought and weakens argument construction — related
• Both ChatGPT feedback and peer feedback prompted doctoral students to actively exercise hard evaluative judgement in statistical analysis, through different challenges — a narrower finding that bears on this claim
• In a small case study, students given ChatGPT feedback valued its immediacy and alternative analytical perspectives, but one distrusted its answers as 'often made up' for complex data analysis — a narrower finding that bears on this claim
• ChatGPT feedback required doctoral students to validate the procedural accuracy of algorithm-generated outputs, with judgement shaped by prompt quality — a narrower finding that bears on this claim
• Meta-reviews reach contradictory conclusions on whether AI reliance harms or helps critical thinking, with substantial design limitations limiting generalizability — related
• Undergraduate students’ genAI knowledge contains specific confident misconceptions, most prevalently that AI chatbots perform web searches — related
• High-level feedback on reasoning and synthesis should remain a human responsibility for the foreseeable future — related
• LLM responses that are incomplete or incorrect prompt students to question, verify and improve the information, fostering critical thinking through cycles of validation and correction — reports the opposite
• LLMs perform worse on complex, open-ended, or subjective assignments than on short, clearly formulated answers — related
• Retrieval-augmented generation reduces but does not eliminate hallucinations in AI-assisted verification — related
• Task design balancing challenge and accessibility determines the success of AI-powered collaborative learning environments — related
• AI solutions can automate routine tasks but cannot yet handle the complex tasks of higher learning — related
• Documented GenAI-related challenges to active learning include cognitive debt, metacognitive laziness, unreflected acceptance, and overreliance on GenAI access — possibly the same claim (merge candidate)
• Generative AI platforms cannot yet independently lead meaningful innovation; at best they augment human intelligence for innovation — related
• The convenience and low opportunity cost of LLMs poses a risk of over-reliance, deskilling, and reduced judgment competence when learners outsource tasks that are essential learning opportunities. — related
• Students reported ChatGPT limitations in Quantum Theory learning: insufficient in-depth explanations, inability to interpret graphs or graphic equations, and occasional unexplainable mathematical errors — a narrower finding that bears on this claim
• Students found using a chatbot to solve a photoelectric-effect problem and evaluating its answer insightful, and the authors propose group checking as a way to verify chatbot answers — related
• Systematic prompting of an LLM can generate cognitively activating formative feedback aligned with research-based scaffold types (demonstrated, not yet empirically validated) — related
• Students most often report hybrid interaction modes with generative AI, with passive task delegation and verification-only use as smaller patterns
• Leading LLMs hallucinated legal authorities in 58%-88% of tested cases while failing to recognise incorrect legal assumptions in prompts
• AI-directed problem selection may diminish the metacognitive dimension of self-testing while preserving retrieval practice — related
• A first evaluation of 3B confirms its conformance to critical usability requirements including safety and comfort, believability, simplicity, acceptability, extensibility, affordability, and mobility — related
• AI functioned primarily as a source of preliminary ideas and cognitive support rather than as a substitute for professional judgment, with students identifying contextual limitations of AI recommendations — a narrower finding that bears on this claim
• An AI assistant configured on the ISLE approach can scaffold the epistemic core of inquiry, at the level of feasibility — related
• AI roles in learning communities risk passivity, diminished authenticity, and overdependence if reflective inquiry is bypassed — related
• Mathematical hallucinations were rare but teachers used professional knowledge to adjust mathematical properties — a narrower finding that bears on this claim
• GenAI's default output on a take-home exam question remained at the quantitative, multistructural SOLO phase, producing polished but pedagogically thin responses — related
• Peer feedback promoted critical reflection, collaboration and engagement, but students doubted its reliability because peers were statistics beginners — related
• Within prompt–response AI interfaces, students without guidance rarely move beyond prompt tuning, yielding interactions of limited educational value and increased reliance on outputs — related
• Human oversight remained necessary alongside AI-based tutoring, especially for misconceptions and advanced topics — a narrower finding that bears on this claim
• Judge-condition verification did not consistently improve LLM feedback quality and often propagated Peer agents' errors — a narrower finding that bears on this claim
• Trust in AI varied by positioning: partners saw bias and assigned their own grades, while substitutes trusted AI scores as actual grades — related
• Conversational synthesis can hide source plurality behind one voice, weakening verification relative to search — related
• Delegating intellectual work to generative AI risks eroding epistemic agency through reduced critical effort and cognitive offloading — related
• Automated AI assessment systems reduce grading time and improve evaluation consistency in nursing education — related
• Teacher-provided correct answers and pre-prompting reduce erroneous LLM output and hallucinations compared with open chatbot use — related
• LLM feedback agents show model-specific over-rejection of valid-alternative reasoning and over-validation of incorrect solutions — related
• Pseudo-agency: students may appear engaged without understanding the epistemic purpose of their actions — related
• Thematic analysis of open responses surfaced four themes: immediacy as core value, calibrated trust, pinpointing the mistake, and progress and control — related
• In the pilot workshop, the LLM unreliably produced fill-in-the-blank snippets, often returning complete code solutions instead — a narrower finding that bears on this claim
• At the implementation checkpoint, several groups added an explicit human-approval step after encountering local-LLM structured-output reliability limitations — a narrower finding that bears on this claim
• GenAI functions as productive self-scaffolding when students retain executive control, and becomes unproductive when reasoning is outsourced — related
• A persistent theory-practice gap: students may develop conceptual AI understanding but lack opportunities to apply it in authentic contexts — related
• Synergy is more likely in simpler content generation tasks and when humans exceed AI performance — related
• AI takeover of metacognitive monitoring may foster illusion of competence (untested conjecture) — related
• AWE feedback has limited depth and questionable accuracy for evaluating complex writing aspects — related
• GenAI use in creative contexts tends toward regression to the mean: less skilled users improve while higher skilled users decline — related
• Students evolved from passive users of GenAI to critical evaluators, developing strategies for bias detection and source validation — a narrower finding that bears on this claim
• Journal analysis shows heavy GenAI reliance in secondary research and transcription but student autonomy in planning and interpretation of primary research — related
• Optimizing prompts can vastly improve the accuracy of LLM outputs, but critical evaluation of the resulting outputs still requires domain expertise. — related
• A general-purpose LLM used as a tutor without pedagogical prompting overwhelmingly provides high assistance, with more than 95% of responses classified as Explaining or Solving — related
• Generative AI generates responses based on probability and pattern prediction, not reasoning or understanding — related
• The IPS evidence base predominantly addresses information search and selection, with library and ecological-validity biases limiting generalizability to broader IPS competence — related
• AI over-reliance can produce inversion effects, reducing cognitive processing depth and learning outcomes — related
• On novel, hard tasks, superior trained adults tend to jump at conclusions, accept results without surety, and become confused, more than orthodox psychology supposes — a narrower finding that bears on this claim