  The objective of lifelong reinforcement learning (RL) is to optimize agents
which can continuously adapt and interact in changing environments. However,
current RL approaches fail drastically when environments are non-stationary and
interactions are non-episodic. We propose Lifelong Skill Planning (LiSP), an
algorithmic framework for non-episodic lifelong RL based on planning in an
abstract space of higher-order skills. We learn the skills in an unsupervised
manner using intrinsic rewards and plan over the learned skills using a learned
dynamics model. Moreover, our framework permits skill discovery even from
offline data, thereby reducing the need for excessive real-world interactions.
We demonstrate empirically that LiSP successfully enables long-horizon planning
and learns agents that can avoid catastrophic failures even in challenging
non-stationary and non-episodic environments derived from gridworld and MuJoCo
benchmarks.
