  A practical approach to robot reinforcement learning is to first collect a
large batch of real or simulated robot interaction data, using some data
collection policy, and then learn from this data to perform various tasks,
using offline learning algorithms. Previous work focused on manually designing
the data collection policy, and on tasks where suitable policies can easily be
designed, such as random picking policies for collecting data about object
grasping. For more complex tasks, however, it may be difficult to find a data
collection policy that explores the environment effectively, and produces data
that is diverse enough for the downstream task. In this work, we propose that
data collection policies should actively explore the environment to collect
diverse data. In particular, we develop a simple-yet-effective goal-conditioned
reinforcement-learning method that actively focuses data collection on novel
observations, thereby collecting a diverse data-set. We evaluate our method on
simulated robot manipulation tasks with visual inputs and show that the
improved diversity of active data collection leads to significant improvements
in the downstream learning tasks.
