  In this article, we aim to provide a literature review of different
formulations and approaches to continual reinforcement learning (RL), also
known as lifelong or non-stationary RL. We begin by discussing our perspective
on why RL is a natural fit for studying continual learning. We then provide a
taxonomy of different continual RL formulations by mathematically
characterizing two key properties of non-stationarity, namely, the scope and
driver non-stationarity. This offers a unified view of various formulations.
Next, we review and present a taxonomy of continual RL approaches. We go on to
discuss evaluation of continual RL agents, providing an overview of benchmarks
used in the literature and important metrics for understanding agent
performance. Finally, we highlight open problems and challenges in bridging the
gap between the current state of continual RL and findings in neuroscience.
While still in its early days, the study of continual RL has the promise to
develop better incremental reinforcement learners that can function in
increasingly realistic applications where non-stationarity plays a vital role.
These include applications such as those in the fields of healthcare,
education, logistics, and robotics.
