  Generalization is a pivotal challenge for agents following natural language
instructions. To approach this goal, we leverage a vision-language model (VLM)
for visual grounding and transfer its vision-language knowledge into
reinforcement learning (RL) for object-centric tasks, which makes the agent
capable of zero-shot generalization to unseen objects and instructions. By
visual grounding, we obtain an object-grounded confidence map for the target
object indicated in the instruction. Based on this map, we introduce two routes
to transfer VLM knowledge into RL. Firstly, we propose an object-grounded
intrinsic reward function derived from the confidence map to more effectively
guide the agent towards the target object. Secondly, the confidence map offers
a more unified, accessible task representation for the agent's policy, compared
to language embeddings. This enables the agent to process unseen objects and
instructions through comprehensible visual confidence maps, facilitating
zero-shot object-level generalization. Single-task experiments prove that our
intrinsic reward significantly improves performance on challenging skill
learning. In multi-task experiments, through testing on tasks beyond the
training set, we show that the agent, when provided with the confidence map as
the task representation, possesses better generalization capabilities than
language-based conditioning. The code is available at
https://github.com/PKU-RL/COPL.
