  Robots operating in human-centric environments require the integration of
visual grounding and grasping capabilities to effectively manipulate objects
based on user instructions. This work focuses on the task of referring grasp
synthesis, which predicts a grasp pose for an object referred through natural
language in cluttered scenes. Existing approaches often employ multi-stage
pipelines that first segment the referred object and then propose a suitable
grasp, and are evaluated in private datasets or simulators that do not capture
the complexity of natural indoor scenes. To address these limitations, we
develop a challenging benchmark based on cluttered indoor scenes from OCID
dataset, for which we generate referring expressions and connect them with
4-DoF grasp poses. Further, we propose a novel end-to-end model (CROG) that
leverages the visual grounding capabilities of CLIP to learn grasp synthesis
directly from image-text pairs. Our results show that vanilla integration of
CLIP with pretrained models transfers poorly in our challenging benchmark,
while CROG achieves significant improvements both in terms of grounding and
grasping. Extensive robot experiments in both simulation and hardware
demonstrate the effectiveness of our approach in challenging interactive object
grasping scenarios that include clutter.
