  Humans can quickly learn new behaviors by leveraging background world
knowledge. In contrast, agents trained with reinforcement learning (RL)
typically learn behaviors from scratch. We thus propose a novel approach that
uses the vast amounts of general and indexable world knowledge encoded in
vision-language models (VLMs) pre-trained on Internet-scale data for embodied
RL. We initialize policies with VLMs by using them as promptable
representations: embeddings that encode semantic features of visual
observations based on the VLM's internal knowledge and reasoning capabilities,
as elicited through prompts that provide task context and auxiliary
information. We evaluate our approach on visually-complex, long horizon RL
tasks in Minecraft and robot navigation in Habitat. We find that our policies
trained on embeddings from off-the-shelf, general-purpose VLMs outperform
equivalent policies trained on generic, non-promptable image embeddings. We
also find our approach outperforms instruction-following methods and performs
comparably to domain-specific embeddings. Finally, we show that our approach
can use chain-of-thought prompting to produce representations of common-sense
semantic reasoning, improving policy performance in novel scenes by 1.5 times.
