  We focus on the task of language-conditioned grasping in clutter, in which a
robot is supposed to grasp the target object based on a language instruction.
Previous works separately conduct visual grounding to localize the target
object, and generate a grasp for that object. However, these works require
object labels or visual attributes for grounding, which calls for handcrafted
rules in planner and restricts the range of language instructions. In this
paper, we propose to jointly model vision, language and action with
object-centric representation. Our method is applicable under more flexible
language instructions, and not limited by visual grounding error. Besides, by
utilizing the powerful priors from the pre-trained multi-modal model and grasp
model, sample efficiency is effectively improved and the sim2real problem is
relived without additional data for transfer. A series of experiments carried
out in simulation and real world indicate that our method can achieve better
task success rate by less times of motion under more flexible language
instructions. Moreover, our method is capable of generalizing better to
scenarios with unseen objects and language instructions. Our code is available
at https://github.com/xukechun/Vision-Language-Grasping
