Publikation
Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning
Ahmed Hendawy; Henrik Metternich; Théo Vincent; Mahdi Kallel; Jan Peters; Carlo D'Eramo
In: Carl Vondrick; Bharath Hariharan; Colin Raffel; Lerrel Pinto; Diyi Yang; Aleksandra Faust (Hrsg.). The Fourteenth International Conference on Learning Representations, ICLR 2026, Rio de Janeiro, Brazil, April 23-27, 2026. International Conference on Learning Representations (ICLR), Pages 1-47, proceedings.iclr.cc, 2026.
Zusammenfassung
The use of target networks is a popular approach for estimating value functions in
deep Reinforcement Learning (RL). While effective, the target network remains a
compromise solution that preserves stability at the cost of slowly moving targets,
thus delaying learning. Conversely, using the online network as a bootstrapped tar-
get is intuitively appealing, albeit well-known to lead to unstable learning. In this
work, we aim to obtain the best out of both worlds by introducing a novel update
rule that computes the target using the MINimum estimate between the Target
and Online network, giving rise to our method, MINTO. Through this simple, yet
effective modification, we show that MINTO enables faster and stable value func-
tion learning, by mitigating the potential overestimation bias of using the online
network for bootstrapping. Notably, MINTO can be seamlessly integrated into a
wide range of value-based and actor-critic algorithms with a negligible cost. We
evaluate MINTO extensively across diverse benchmarks, spanning online and of-
fline RL, as well as discrete and continuous action spaces. Across all benchmarks,
MINTO consistently improves performance, demonstrating its broad applicability
and effectiveness.