  The ability to grasp objects in-the-wild from open-ended language
instructions constitutes a fundamental challenge in robotics. An open-world
grasping system should be able to combine high-level contextual with low-level
physical-geometric reasoning in order to be applicable in arbitrary scenarios.
Recent works exploit the web-scale knowledge inherent in large language models
(LLMs) to plan and reason in robotic context, but rely on external vision and
action models to ground such knowledge into the environment and parameterize
actuation. This setup suffers from two major bottlenecks: a) the LLM's
reasoning capacity is constrained by the quality of visual grounding, and b)
LLMs do not contain low-level spatial understanding of the world, which is
essential for grasping in contact-rich scenarios. In this work we demonstrate
that modern vision-language models (VLMs) are capable of tackling such
limitations, as they are implicitly grounded and can jointly reason about
semantics and geometry. We propose OWG, an open-world grasping pipeline that
combines VLMs with segmentation and grasp synthesis models to unlock grounded
world understanding in three stages: open-ended referring segmentation,
grounded grasp planning and grasp ranking via contact reasoning, all of which
can be applied zero-shot via suitable visual prompting mechanisms. We conduct
extensive evaluation in cluttered indoor scene datasets to showcase OWG's
robustness in grounding from open-ended language, as well as open-world robotic
grasping experiments in both simulation and hardware that demonstrate superior
performance compared to previous supervised and zero-shot LLM-based methods.
