  Learning tabula rasa, that is without any prior knowledge, is the prevalent
workflow in reinforcement learning (RL) research. However, RL systems, when
applied to large-scale settings, rarely operate tabula rasa. Such large-scale
systems undergo multiple design or algorithmic changes during their development
cycle and use ad hoc approaches for incorporating these changes without
re-training from scratch, which would have been prohibitively expensive.
Additionally, the inefficiency of deep RL typically excludes researchers
without access to industrial-scale resources from tackling
computationally-demanding problems. To address these issues, we present
reincarnating RL as an alternative workflow or class of problem settings, where
prior computational work (e.g., learned policies) is reused or transferred
between design iterations of an RL agent, or from one RL agent to another. As a
step towards enabling reincarnating RL from any agent to any other agent, we
focus on the specific setting of efficiently transferring an existing
sub-optimal policy to a standalone value-based RL agent. We find that existing
approaches fail in this setting and propose a simple algorithm to address their
limitations. Equipped with this algorithm, we demonstrate reincarnating RL's
gains over tabula rasa RL on Atari 2600 games, a challenging locomotion task,
and the real-world problem of navigating stratospheric balloons. Overall, this
work argues for an alternative approach to RL research, which we believe could
significantly improve real-world RL adoption and help democratize it further.
Open-sourced code and trained agents at
https://agarwl.github.io/reincarnating_rl.
