  Benefiting from language flexibility and compositionality, humans naturally
intend to use language to command an embodied agent for complex tasks such as
navigation and object manipulation. In this work, we aim to fill the blank of
the last mile of embodied agents -- object manipulation by following human
guidance, e.g., "move the red mug next to the box while keeping it upright." To
this end, we introduce an Automatic Manipulation Solver (AMSolver) system and
build a Vision-and-Language Manipulation benchmark (VLMbench) based on it,
containing various language instructions on categorized robotic manipulation
tasks. Specifically, modular rule-based task templates are created to
automatically generate robot demonstrations with language instructions,
consisting of diverse object shapes and appearances, action types, and motion
constraints. We also develop a keypoint-based model 6D-CLIPort to deal with
multi-view observations and language input and output a sequence of 6 degrees
of freedom (DoF) actions. We hope the new simulator and benchmark will
facilitate future research on language-guided robotic manipulation.
