  Reinforcement learning systems have the potential to enable continuous
improvement in unstructured environments, leveraging data collected
autonomously. However, in practice these systems require significant amounts of
instrumentation or human intervention to learn in the real world. In this work,
we propose a system for reinforcement learning that leverages multi-task
reinforcement learning bootstrapped with prior data to enable continuous
autonomous practicing, minimizing the number of resets needed while being able
to learn temporally extended behaviors. We show how appropriately provided
prior data can help bootstrap both low-level multi-task policies and strategies
for sequencing these tasks one after another to enable learning with minimal
resets. This mechanism enables our robotic system to practice with minimal
human intervention at training time while being able to solve long horizon
tasks at test time. We show the efficacy of the proposed system on a
challenging kitchen manipulation task both in simulation and in the real world,
demonstrating the ability to practice autonomously in order to solve temporally
extended problems.
