  We study learning control in an online reset-free lifelong learning scenario,
where mistakes can compound catastrophically into the future and the underlying
dynamics of the environment may change. Traditional model-free policy learning
methods have achieved successes in difficult tasks due to their broad
flexibility, but struggle in this setting, as they can activate failure modes
early in their lifetimes which are difficult to recover from and face
performance degradation as dynamics change. On the other hand, model-based
planning methods learn and adapt quickly, but require prohibitive levels of
computational resources. We present a new algorithm, Adaptive Online Planning
(AOP), that achieves strong performance in this setting by combining
model-based planning with model-free learning. By approximating the uncertainty
of the model-free components and the planner performance, AOP is able to call
upon more extensive planning only when necessary, leading to reduced
computation times, while still gracefully adapting behaviors in the face of
unpredictable changes in the world -- even when traditional RL fails.
