Phase VI AI-enabled systems face six documented challenges: explainability, hallucination risk, prompt sensitivity, computational cost, validation complexity, and limited large-scale evidence¶

Claim · All claims
Evidence · 1 study · review r2 · q1
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q1 i? Contemporary AI-enabled serious game systems remain nascent, with three reliability challenges (explainability, hallucination risk, prompt sensitivity) and three system-level deployment challenges (computational cost, validation complexity, limited large-scale evidence). → Tripathi 2026

Evidence¶

Tripathi 2026¶

Tripathi, P., & Kapralos, B. (2026). AI-Enabled Serious Games: Integrating Intelligence and Adaptivity in Training Systems. arXiv. https://arxiv.org/abs/2605.21962 doi:10.48550/arxiv.2605.21962

q1 · i? · review · r2

The chapter identifies six challenges for Phase VI systems, citing prior work. Neural models produce predictions without showing their reasoning; LLMs sometimes generate factually incorrect content without built-in flagging; prompt sensitivity creates reproducibility challenges; computational cost and economic viability at scale have not been demonstrated; traditional software testing is insufficient for dynamically generating systems; and rigorous controlled studies measuring skill transfer over time remain scarce.

"most Phase VI systems remain research prototypes; rigorous controlled studies measuring skill transfer over time remain scarce [94]."

Discussion¶

Related Claims¶

• LLM-assisted inductive qualitative coding carries risks of superficial themes, broad or redundant codes, and hallucinated interpretations, so LLMs should augment rather than replace human researchers — related
• Training large AI models carries substantial environmental and human labor costs — related
• Zero-shot LLM errors follow four recurring patterns: over-interpretation, failure to detect relevant information, hallucination, and failure to generate a response — related
• Teachers who perceive the challenges and barriers of using AI as high (cost) may be less motivated to use it (theoretical argument). — related
• Few AIED 2025 papers report computational costs or discuss environmental impacts, and reporting is non-standardized — related
• AWE feedback has limited depth and questionable accuracy for evaluating complex writing aspects — related
• The most reported risks of AI tools in science and chemistry education are ethical issues, including gender and racial bias, hallucinations, copyright infringement, and plagiarism — related
• CA prototype technologies shifted from rule-based and deterministic systems toward open-source tools, pre-trained LLMs, and retrieval-augmented generation pipelines — related
• Generative AI-generated hints can be as effective as human-generated hints for mathematics performance, but ensuring accuracy requires validation procedures that discard a substantial share of generated hints — a narrower finding that bears on this claim
• The system has no empirical evaluation yet: profile accuracy has not been assessed and learning outcomes remain an open question — related
• Across 28 project contexts, human-in-the-loop and independent peer review identified no conceptual or mathematical hallucinations, and Python visualization scripts executed without compilation errors on the zero-shot attempt. — related
• Recurring risks of GenAI in STEAM education include hallucination, bias, superficial completion strategies, and compromised assessment validity — related
• Students reported challenges with generated-model scale fidelity, shared attention in multi-user XR, motion comfort, and control over generation — related
• Three recurring AI failure modes arose in this project-based learning context: plausible-but-incorrect code, missing specialized knowledge, and limited long-term context — related
• Gen-AI hallucinates sources and values and produces convincing but incorrect explanations, risking student misinformation — a narrower finding that bears on this claim
• Qualitative error analysis yields five recurring error types, with over-inference most prevalent (33.5%), followed by evidence misidentification (23.4%) and hallucination (19.8%) — related
• Evidence on LLM-based family education remains limited across families and educational purposes, especially on sustained personalization, repair labor, and negotiation of authority and rules — a narrower finding that bears on this claim
• Reported limitation profiles differ by automation method, with prompt sensitivity especially frequent in LLM papers — related
• Many agentic multi-agent implementations remain at prototype or conceptual stage with limited large-scale deployment and empirical validation — a narrower finding that bears on this claim
• GenAI raises privacy, transparency, bias and accuracy concerns in group work, though empirical evidence remains limited — related
• Teacher-AI design collaboration faces recurring challenges including prompt literacy, curriculum misalignment, epistemic uncertainty, and explainability barriers — related
• Switching from reference-based to contextual evaluation prompts improved GenAI grading accuracy — a narrower finding that bears on this claim
• LLMs in educational applications hallucinate in 15–25% of responses and can amplify bias against non-native language inputs — related
• Retrieval-augmented generation reduces but does not eliminate hallucinations in AI-assisted verification — related
• Open issues for dynamic assessment include cost-effectiveness, training, reliability and validity, and generalization — related
• Large language models can support learning and teaching across education levels, from reading and writing support for elementary students to professional training — related
• Systematic prompting of an LLM can generate cognitively activating formative feedback aligned with research-based scaffold types (demonstrated, not yet empirically validated) — related
• AI in education poses challenges across three interrelated domains: knowledge itself, the process of knowing, and the socio-environmental impact of knowledge production — related
• Research on AI support for EL literacy remains limited, narrow in scope, and lacking for diverse EL subgroups — a broader claim this one bears on
• AI-based assessment practices within PBL contexts remain underdeveloped, with limited empirical evidence on automated feedback, formative assessment, and learning analytics — related
• Open questions for learnity graphs include validation, legitimacy, grading avoidance, algorithmic guidance balance, and privacy — related
• LLM-based simulated students surpass rule-based systems in natural language generation and situational flexibility, but concerns about bias and evaluation reliability persist — related
• Regulatory and ethics compliance posed substantial, underestimated challenges that delayed the ITS rollout by a year — related
• GenAI models span multiple agent types, mainly model-based and goal-based, but categorisation is difficult without model specifics — related