  Grounding language to the visual observations of a navigating agent can be
performed using off-the-shelf visual-language models pretrained on
Internet-scale data (e.g., image captions). While this is useful for matching
images to natural language descriptions of object goals, it remains disjoint
from the process of mapping the environment, so that it lacks the spatial
precision of classic geometric maps. To address this problem, we propose
VLMaps, a spatial map representation that directly fuses pretrained
visual-language features with a 3D reconstruction of the physical world. VLMaps
can be autonomously built from video feed on robots using standard exploration
approaches and enables natural language indexing of the map without additional
labeled data. Specifically, when combined with large language models (LLMs),
VLMaps can be used to (i) translate natural language commands into a sequence
of open-vocabulary navigation goals (which, beyond prior work, can be spatial
by construction, e.g., "in between the sofa and TV" or "three meters to the
right of the chair") directly localized in the map, and (ii) can be shared
among multiple robots with different embodiments to generate new obstacle maps
on-the-fly (by using a list of obstacle categories). Extensive experiments
carried out in simulated and real world environments show that VLMaps enable
navigation according to more complex language instructions than existing
methods. Videos are available at https://vlmaps.github.io.
