  To achieve general artificial intelligence, reinforcement learning (RL)
agents should learn not only to optimize returns for one specific task but also
to constantly build more complex skills and scaffold their knowledge about the
world, without forgetting what has already been learned. In this paper, we
discuss the desired characteristics of environments that can support the
training and evaluation of lifelong reinforcement learning agents, review
existing environments from this perspective, and propose recommendations for
devising suitable environments in the future.
