  A key challenge in lifelong reinforcement learning (RL) is the loss of
plasticity, where previous learning progress hinders an agent's adaptation to
new tasks. While regularization and resetting can help, they require precise
hyperparameter selection at the outset and environment-dependent adjustments.
Building on the principled theory of online convex optimization, we present a
parameter-free optimizer for lifelong RL, called TRAC, which requires no tuning
or prior knowledge about the distribution shifts. Extensive experiments on
Procgen, Atari, and Gym Control environments show that TRAC works surprisingly
well-mitigating loss of plasticity and rapidly adapting to challenging
distribution shifts-despite the underlying optimization problem being nonconvex
and nonstationary.
