  We introduce Wonderful Team, a multi-agent visual LLM (VLLM) framework for
solving robotics problems in the zero-shot regime. By zero-shot we mean that,
for a novel environment, we feed a VLLM an image of the robot's environment and
a description of the task, and have the VLLM output the sequence of actions
necessary for the robot to complete the task. Prior work on VLLMs in robotics
has largely focused on settings where some part of the pipeline is fine-tuned,
such as tuning an LLM on robot data or training a separate vision encoder for
perception and action generation. Surprisingly, due to recent advances in the
capabilities of VLLMs, this type of fine-tuning may no longer be necessary for
many tasks. In this work, we show that with careful engineering, we can prompt
a single off-the-shelf VLLM to handle all aspects of a robotics task, from
high-level planning to low-level location-extraction and action-execution.
Wonderful Team builds on recent advances in multi-agent LLMs to partition tasks
across an agent hierarchy, making it self-corrective and able to effectively
partition and solve even long-horizon tasks. Extensive experiments on VIMABench
and real-world robotic environments demonstrate the system's capability to
handle a variety of robotic tasks, including manipulation, visual
goal-reaching, and visual reasoning, all in a zero-shot manner. These results
underscore a key point: vision-language models have progressed rapidly in the
past year, and should strongly be considered as a backbone for robotics
problems going forward.
