  Recent advances in vision-language models (VLMs) have led to improved
performance on tasks such as visual question answering and image captioning.
Consequently, these models are now well-positioned to reason about the physical
world, particularly within domains such as robotic manipulation. However,
current VLMs are limited in their understanding of the physical concepts (e.g.,
material, fragility) of common objects, which restricts their usefulness for
robotic manipulation tasks that involve interaction and physical reasoning
about such objects. To address this limitation, we propose PhysObjects, an
object-centric dataset of 39.6K crowd-sourced and 417K automated physical
concept annotations of common household objects. We demonstrate that
fine-tuning a VLM on PhysObjects improves its understanding of physical object
concepts, including generalization to held-out concepts, by capturing human
priors of these concepts from visual appearance. We incorporate this physically
grounded VLM in an interactive framework with a large language model-based
robotic planner, and show improved planning performance on tasks that require
reasoning about physical object concepts, compared to baselines that do not
leverage physically grounded VLMs. We additionally illustrate the benefits of
our physically grounded VLM on a real robot, where it improves task success
rates. We release our dataset and provide further details and visualizations of
our results at https://iliad.stanford.edu/pg-vlm/.
