  Legged robots are physically capable of navigating a diverse variety of
environments and overcoming a wide range of obstructions. For example, in a
search and rescue mission, a legged robot could climb over debris, crawl
through gaps, and navigate out of dead ends. However, the robot's controller
needs to respond intelligently to such varied obstacles, and this requires
handling unexpected and unusual scenarios successfully. This presents an open
challenge to current learning methods, which often struggle with generalization
to the long tail of unexpected situations without heavy human supervision. To
address this issue, we investigate how to leverage the broad knowledge about
the structure of the world and commonsense reasoning capabilities of
vision-language models (VLMs) to aid legged robots in handling difficult,
ambiguous situations. We propose a system, VLM-Predictive Control (VLM-PC),
combining two key components that we find to be crucial for eliciting
on-the-fly, adaptive behavior selection with VLMs: (1) in-context adaptation
over previous robot interactions and (2) planning multiple skills into the
future and replanning. We evaluate VLM-PC on several challenging real-world
obstacle courses, involving dead ends and climbing and crawling, on a Go1
quadruped robot. Our experiments show that by reasoning over the history of
interactions and future plans, VLMs enable the robot to autonomously perceive,
navigate, and act in a wide range of complex scenarios that would otherwise
require environment-specific engineering or human guidance.
