Hyperparameter sweeps reproduce the lesson's claims: γ = 0 never solves the task while ε = 1 still converges¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? In the exploration and learning-rate sweeps, all tested ε values including ε = 1 eventually reach the optimal return, and all tested α values converge on this small problem. → Jueng 2026 q2 i? In the discount sweep, γ ∈ {0.5, 0.95} reach the optimum while γ = 0 never solves the task, with a final return of −6.1. → Jueng 2026 (2)

Evidence¶

Jueng 2026¶

Jueng, Ekkachai. (2026). Q-Learning Lab: Teaching Reinforcement Learning Through Learner-Generated Trace Analysis. https://arxiv.org/abs/2607.10802 doi:10.48550/arxiv.2607.10802

q2 · i? · causal · r2

Multi-seed numerical sweep experiment: for each setting the reference implementation trains for 600 episodes and evaluates the greedy policy every 20 episodes by undiscounted return from the start state, aggregating 10 random seeds. All tested ε values 'eventually reach the optimal return—includingε= 1'.

"Allε∈{0.05,0.3,1.0}eventually reach the optimal return—includingε= 1, a nice classroom demonstration that Q-learning is off-policy and learns the greedy policy even from purely random behavior, though with slower, noisier progress."

Jueng 2026 (2)¶

Jueng, Ekkachai. (2026). Q-Learning Lab: Teaching Reinforcement Learning Through Learner-Generated Trace Analysis. https://arxiv.org/abs/2607.10802. doi:10.48550/arxiv.2607.10802

q2 · i? · causal · r2

Same 600-episode, 10-seed sweep experiment, discount factor arm. The article reports that 'γ= 0never solves the task (final return−6.1)' while γ ∈ {0.5, 0.95} reach the optimum of 9.3.

"The discount sweep is the starkest:γ∈{0.5,0.95}reach the optimum, whileγ= 0never solves the task (final return−6.1)—the myopic agent cannot propagate the goal reward backwards, exactly the “short-sighted agent” the lesson describes."

Discussion¶

Related Claims¶

• OptimNN is insensitive to optimizer learning rate and network hyperparameters, unlike plain SGD on BKT — related
• The tool's Q-learning matches a value-iteration ground truth in 98.5% of non-terminal states on average — related
• Editing the pit reward produces two behaviorally identical but diagnostically opposite failure modes, separated by a threshold at 7.774 — related
• ProPRL is sensitive to learning rate but comparatively insensitive to its structural hyperparameters, with propagation depth around 5 capturing most multi-hop benefit — related
• Iterate averaging with step sizes decreasing slower than O(1/n) converges at an optimal rate — related
• Many learning algorithms, including neural network training and Q-learning, have recursive forms of the stochastic approximation type — related