  Vision-Language Models (VLMs) have shown great success as foundational models
for downstream vision and natural language applications in a variety of
domains. However, these models lack the spatial understanding necessary for
robotics applications where the agent must reason about the affordances
provided by the 3D world around them. We present a system which trains on
spatially-localized egocentric videos in order to connect visual input and task
descriptions to predict a task's spatial affordance, that is the location where
a person would go to accomplish the task. We show our approach outperforms the
baseline of using a VLM to map similarity of a task's description over a set of
location-tagged images. Our learning-based approach has less error both on
predicting where a task may take place and on predicting what tasks are likely
to happen at the current location. The resulting system enables robots to use
egocentric sensing to navigate to physical locations of novel tasks specified
in natural language.
