  Reinforcement learning provides a general framework for flexible decision
making and control, but requires extensive data collection for each new task
that an agent needs to learn. In other machine learning fields, such as natural
language processing or computer vision, pre-training on large, previously
collected datasets to bootstrap learning for new tasks has emerged as a
powerful paradigm to reduce data requirements when learning a new task. In this
paper, we ask the following question: how can we enable similarly useful
pre-training for RL agents? We propose a method for pre-training behavioral
priors that can capture complex input-output relationships observed in
successful trials from a wide range of previously seen tasks, and we show how
this learned prior can be used for rapidly learning new tasks without impeding
the RL agent's ability to try out novel behaviors. We demonstrate the
effectiveness of our approach in challenging robotic manipulation domains
involving image observations and sparse reward functions, where our method
outperforms prior works by a substantial margin.
