Publikation
Gradient Iterated Temporal-Difference Learning
Théo Vincent; Kevin Gerhardt; Yogesh Tripathi; Habib Maraqten; Adam White; Martha White; Jan Peters; Carlo D'Eramo
In: Computing Research Repository eprint Journal (CoRR), Vol. abs/2603.07833, Pages 1-33, arXiv, 2026.
Zusammenfassung
Temporal-difference (TD) learning is highly effective at controlling and evaluating an
agent’s long-term outcomes. Most approaches in this paradigm implement a semi-gradient
update to boost the learning speed, which consists of ignoring the gradient of the bootstrapped
estimate. While popular, this type of update is prone to divergence, as Baird’s counterexample
illustrates. Gradient TD methods were introduced to overcome this issue, but have not been
widely used, potentially due to issues with learning speed compared to semi-gradient methods.
Recently, iterated TD learning was developed to increase the learning speed of TD methods.
For that, it learns a sequence of action-value functions in parallel, where each function is op-
timized to represent the application of the Bellman operator over the previous function in the
sequence. While promising, this algorithm can be unstable due to its semi-gradient nature, as
each function tracks a moving target. In this work, we modify iterated TD learning by com-
puting the gradients over those moving targets, aiming to build a powerful gradient TD method
that competes with semi-gradient methods. Our evaluation reveals that this algorithm, called
Gradient Iterated Temporal-Difference learning, has a competitive learning speed against semi-
gradient methods across various benchmarks, including Atari games, a result that no prior work
on gradient TD methods has demonstrated.