Publication
Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization
Daniel Palenicek; Florian Vogt; Jan Peters
In: Computing Research Repository eprint Journal (CoRR), Vol. abs/2502.07523, Pages 1-23, arXiv, 2025.
Abstract
Reinforcement learning has achieved significant milestones, but sample efficiency
remains a bottleneck for real-world applications. Recently, CrossQ has demon-
strated state-of-the-art sample efficiency with a low update-to-data (UTD) ratio of 1.
In this work, we explore CrossQ’s scaling behavior with higher UTD ratios. We
identify challenges in the training dynamics, which are emphasized by higher UTD
ratios. To address these, we integrate weight normalization into the CrossQ frame-
work, a solution that stabilizes training, has been shown to prevent potential loss of
plasticity and keeps the effective learning rate constant. Our proposed approach
reliably scales with increasing UTD ratios, achieving competitive performance
across 25 challenging continuous control tasks on the DeepMind Control Suite
and MyoSuite benchmarks, notably the complex dog and humanoid environments.
This work eliminates the need for drastic interventions, such as network resets, and
offers a simple yet robust pathway for improving sample efficiency and scalability
in model-free reinforcement learning.