  Learning to solve tasks from a sparse reward signal is a major challenge for
standard reinforcement learning (RL) algorithms. However, in the real world,
agents rarely need to solve sparse reward tasks entirely from scratch. More
often, we might possess prior experience to draw on that provides considerable
guidance about which actions and outcomes are possible in the world, which we
can use to explore more effectively for new tasks. In this work, we study how
prior data without reward labels may be used to guide and accelerate
exploration for an agent solving a new sparse reward task. We propose a simple
approach that learns a reward model from online experience, labels the
unlabeled prior data with optimistic rewards, and then uses it concurrently
alongside the online data for downstream policy and critic optimization. This
general formula leads to rapid exploration in several challenging sparse-reward
domains where tabula rasa exploration is insufficient, including the AntMaze
domain, Adroit hand manipulation domain, and a visual simulated robotic
manipulation domain. Our results highlight the ease of incorporating unlabeled
prior data into existing online RL algorithms, and the (perhaps surprising)
effectiveness of doing so.
