  Natural language is often the easiest and most convenient modality for humans
to specify tasks for robots. However, learning to ground language to behavior
typically requires impractical amounts of diverse, language-annotated
demonstrations collected on each target robot. In this work, we aim to separate
the problem of what to accomplish from how to accomplish it, as the former can
benefit from substantial amounts of external observation-only data, and only
the latter depends on a specific robot embodiment. To this end, we propose
Video-Language Critic, a reward model that can be trained on readily available
cross-embodiment data using contrastive learning and a temporal ranking
objective, and use it to score behavior traces from a separate reinforcement
learning actor. When trained on Open X-Embodiment data, our reward model
enables 2x more sample-efficient policy training on Meta-World tasks than a
sparse reward only, despite a significant domain gap. Using in-domain data but
in a challenging task generalization setting on Meta-World, we further
demonstrate more sample-efficient training than is possible with prior
language-conditioned reward models that are either trained with binary
classification, use static images, or do not leverage the temporal information
present in video data.
