  For a general-purpose robot to operate in reality, executing a broad range of
instructions across various environments is imperative. Central to the
reinforcement learning and planning for such robotic agents is a generalizable
reward function. Recent advances in vision-language models, such as CLIP, have
shown remarkable performance in the domain of deep learning, paving the way for
open-domain visual recognition. However, collecting data on robots executing
various language instructions across multiple environments remains a challenge.
This paper aims to transfer video-language models with robust generalization
into a generalizable language-conditioned reward function, only utilizing robot
video data from a minimal amount of tasks in a singular environment. Unlike
common robotic datasets used for training reward functions, human
video-language datasets rarely contain trivial failure videos. To enhance the
model's ability to distinguish between successful and failed robot executions,
we cluster failure video features to enable the model to identify patterns
within. For each cluster, we integrate a newly trained failure prompt into the
text encoder to represent the corresponding failure mode. Our
language-conditioned reward function shows outstanding generalization to new
environments and new instructions for robot planning and reinforcement
learning.
