  Teaching robots novel skills with demonstrations via human-in-the-loop data
collection techniques like kinesthetic teaching or teleoperation puts a heavy
burden on human supervisors. In contrast to this paradigm, it is often
significantly easier to provide raw, action-free visual data of tasks being
performed. Moreover, this data can even be mined from video datasets or the
web. Ideally, this data can serve to guide robot learning for new tasks in
novel environments, informing both "what" to do and "how" to do it. A powerful
way to encode both the "what" and the "how" is to infer a well-shaped reward
function for reinforcement learning. The challenge is determining how to ground
visual demonstration inputs into a well-shaped and informative reward function.
We propose a technique Rank2Reward for learning behaviors from videos of tasks
being performed without access to any low-level states and actions. We do so by
leveraging the videos to learn a reward function that measures incremental
"progress" through a task by learning how to temporally rank the video frames
in a demonstration. By inferring an appropriate ranking, the reward function is
able to guide reinforcement learning by indicating when task progress is being
made. This ranking function can be integrated into an adversarial imitation
learning scheme resulting in an algorithm that can learn behaviors without
exploiting the learned reward function. We demonstrate the effectiveness of
Rank2Reward at learning behaviors from raw video on a number of tabletop
manipulation tasks in both simulations and on a real-world robotic arm. We also
demonstrate how Rank2Reward can be easily extended to be applicable to
web-scale video datasets.
