  Developing an agent capable of adapting to unseen environments remains a
difficult challenge in imitation learning. This work presents Adaptive
Return-conditioned Policy (ARP), an efficient framework designed to enhance the
agent's generalization ability using natural language task descriptions and
pre-trained multimodal encoders. Our key idea is to calculate a similarity
between visual observations and natural language instructions in the
pre-trained multimodal embedding space (such as CLIP) and use it as a reward
signal. We then train a return-conditioned policy using expert demonstrations
labeled with multimodal rewards. Because the multimodal rewards provide
adaptive signals at each timestep, our ARP effectively mitigates the goal
misgeneralization. This results in superior generalization performances even
when faced with unseen text instructions, compared to existing text-conditioned
policies. To improve the quality of rewards, we also introduce a fine-tuning
method for pre-trained multimodal encoders, further enhancing the performance.
Video demonstrations and source code are available on the project website:
\url{https://sites.google.com/view/2023arp}.
