  Observing a human demonstrator manipulate objects provides a rich, scalable
and inexpensive source of data for learning robotic policies. However,
transferring skills from human videos to a robotic manipulator poses several
challenges, not least a difference in action and observation spaces. In this
work, we use unlabeled videos of humans solving a wide range of manipulation
tasks to learn a task-agnostic reward function for robotic manipulation
policies. Thanks to the diversity of this training data, the learned reward
function sufficiently generalizes to image observations from a previously
unseen robot embodiment and environment to provide a meaningful prior for
directed exploration in reinforcement learning. We propose two methods for
scoring states relative to a goal image: through direct temporal regression,
and through distances in an embedding space obtained with time-contrastive
learning. By conditioning the function on a goal image, we are able to reuse
one model across a variety of tasks. Unlike prior work on leveraging human
videos to teach robots, our method, Human Offline Learned Distances (HOLD)
requires neither a priori data from the robot environment, nor a set of
task-specific human demonstrations, nor a predefined notion of correspondence
across morphologies, yet it is able to accelerate training of several
manipulation tasks on a simulated robot arm compared to using only a sparse
reward obtained from task completion.
