  Sample efficiency in Reinforcement Learning (RL) has traditionally been
driven by algorithmic enhancements. In this work, we demonstrate that scaling
can also lead to substantial improvements. We conduct a thorough investigation
into the interplay of scaling model capacity and domain-specific RL
enhancements. These empirical findings inform the design choices underlying our
proposed BRO (Bigger, Regularized, Optimistic) algorithm. The key innovation
behind BRO is that strong regularization allows for effective scaling of the
critic networks, which, paired with optimistic exploration, leads to superior
performance. BRO achieves state-of-the-art results, significantly outperforming
the leading model-based and model-free algorithms across 40 complex tasks from
the DeepMind Control, MetaWorld, and MyoSuite benchmarks. BRO is the first
model-free algorithm to achieve near-optimal policies in the notoriously
challenging Dog and Humanoid tasks.
