  Despite groundbreaking progress in reinforcement learning for robotics,
gameplay, and other complex domains, major challenges remain in applying
reinforcement learning to the evolving, open-world problems often found in
critical application spaces. Reinforcement learning solutions tend to
generalize poorly when exposed to new tasks outside of the data distribution
they are trained on, prompting an interest in continual learning algorithms. In
tandem with research on continual learning algorithms, there is a need for
challenge environments, carefully designed experiments, and metrics to assess
research progress. We address the latter need by introducing a framework for
continual reinforcement-learning development and assessment using Lifelong
Learning Explorer (L2Explorer), a new, Unity-based, first-person 3D exploration
environment that can be continuously reconfigured to generate a range of tasks
and task variants structured into complex and evolving evaluation curricula. In
contrast to procedurally generated worlds with randomized components, we have
developed a systematic approach to defining curricula in response to controlled
changes with accompanying metrics to assess transfer, performance recovery, and
data efficiency. Taken together, the L2Explorer environment and evaluation
approach provides a framework for developing future evaluation methodologies in
open-world settings and rigorously evaluating approaches to lifelong learning.
