  Progress in continual reinforcement learning has been limited due to several
barriers to entry: missing code, high compute requirements, and a lack of
suitable benchmarks. In this work, we present CORA, a platform for Continual
Reinforcement Learning Agents that provides benchmarks, baselines, and metrics
in a single code package. The benchmarks we provide are designed to evaluate
different aspects of the continual RL challenge, such as catastrophic
forgetting, plasticity, ability to generalize, and sample-efficient learning.
Three of the benchmarks utilize video game environments (Atari, Procgen,
NetHack). The fourth benchmark, CHORES, consists of four different task
sequences in a visually realistic home simulator, drawn from a diverse set of
task and scene parameters. To compare continual RL methods on these benchmarks,
we prepare three metrics in CORA: Continual Evaluation, Isolated Forgetting,
and Zero-Shot Forward Transfer. Finally, CORA includes a set of performant,
open-source baselines of existing algorithms for researchers to use and expand
on. We release CORA and hope that the continual RL community can benefit from
our contributions, to accelerate the development of new continual RL
algorithms.
