  Deep reinforcement learning methods traditionally struggle with tasks where
environment rewards are particularly sparse. One successful method of guiding
exploration in these domains is to imitate trajectories provided by a human
demonstrator. However, these demonstrations are typically collected under
artificial conditions, i.e. with access to the agent's exact environment setup
and the demonstrator's action and reward trajectories. Here we propose a
two-stage method that overcomes these limitations by relying on noisy,
unaligned footage without access to such data. First, we learn to map unaligned
videos from multiple sources to a common representation using self-supervised
objectives constructed over both time and modality (i.e. vision and sound).
Second, we embed a single YouTube video in this representation to construct a
reward function that encourages an agent to imitate human gameplay. This method
of one-shot imitation allows our agent to convincingly exceed human-level
performance on the infamously hard exploration games Montezuma's Revenge,
Pitfall! and Private Eye for the first time, even if the agent is not presented
with any environment rewards.
