An analytical chapter identifies reward hacking, engagement optimization over learning, and scaffolding dependency as risks of RL policies with poorly specified rewards¶

Claim · All claims
Evidence · 1 study · review r2 · q1
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q1 i? If reward functions and state representations are defined poorly, RL produces adaptive behavior that appears intelligent while being pedagogically hollow, through reward hacking via memorization, engagement optimization over learning, and scaffolding dependency. → Tripathi 2026

Evidence¶

Tripathi 2026¶

Tripathi, P., & Kapralos, B. (2026). AI-Enabled Serious Games: Integrating Intelligence and Adaptivity in Training Systems. arXiv. https://arxiv.org/abs/2605.21962 doi:10.48550/arxiv.2605.21962

q1 · i? · review · r2

The chapter identifies three failure scenarios as analytical risks, citing prior work: an RL policy may learn to present repeated similar scenarios because learners achieve high accuracy through pattern matching; a policy may present easier, more engaging content rather than challenging content producing learning gains; and extensive hints may create dependency where learners perform well with support but fail without it.

"If these are defined poorly, RL produces adaptive behavior that appears intelligent while being pedagogically hollow."

Discussion¶

Related Claims¶

• Personalized Learning Effects Vary With Fidelity — the general claim this finding bears on, with the studies behind it
• AI-supported adaptive systems are reported to enhance adult learner engagement, motivation, and outcomes when aligned with learner goals and prior knowledge — related
• Over eight weeks, university learners receiving reinforcement learning-optimized oral practice sequencing attained normalized learning gains approximately 2.2 times higher than learners following fixed curricula — related
• Expected tangible rewards reduce later free-choice engagement with a rewarded task, while verbal praise does not — related
• Reported outcomes of AI-supported university physics education are mixed and context-dependent, and the evidence does not support a conclusion that AI is uniformly effective — related
• A screenshot-only reward was reward-hacked into producing visually convincing but unplayable games; hardening the reward with an interactivity probe and hard-fail gate reversed the collapse — a narrower finding that bears on this claim
• Edu-QuRater rewards alone raise pedagogical-quality win rate (77.70%) but fall below parity on instruction following (40.54%), consistent with over-optimizing a proxy reward — a narrower finding that bears on this claim
• Reinforcement learning with a learned alignment reward substantially improves question-concept alignment after supervised fine-tuning, while answerability remains high throughout training — related
• An engagement-optimized RL tutor exhibits reward hacking, achieving the highest cumulative reward while producing minimal learning and a 70.8% behavioral-safety violation rate — a narrower finding that bears on this claim
• Editing the pit reward produces two behaviorally identical but diagnostically opposite failure modes, separated by a threshold at 7.774 — a narrower finding that bears on this claim
• Reinforcement learning in adaptive educational systems faces sample inefficiency and reward-design instability, motivating hybrid RL-ML approaches — related
• Using a trained StudentSim as the reward for chess tutor RL yields a tutor that expert human raters score best on accuracy, guidance quality, and personalization among the rewards studied — a narrower finding that bears on this claim
• Among RL papers statistically outperforming baselines, normalized learning gain was the most common effective reward source, used in over half of the papers (n = 10) — related
• Reinforcement learning operationalizes motivational constructs by representing motivation in the reward function — a broader claim this one bears on