  From rearranging objects on a table to putting groceries into shelves, robots
must plan precise action points to perform tasks accurately and reliably. In
spite of the recent adoption of vision language models (VLMs) to control robot
behavior, VLMs struggle to precisely articulate robot actions using language.
We introduce an automatic synthetic data generation pipeline that
instruction-tunes VLMs to robotic domains and needs. Using the pipeline, we
train RoboPoint, a VLM that predicts image keypoint affordances given language
instructions. Compared to alternative approaches, our method requires no
real-world data collection or human demonstration, making it much more scalable
to diverse environments and viewpoints. In addition, RoboPoint is a general
model that enables several downstream applications such as robot navigation,
manipulation, and augmented reality (AR) assistance. Our experiments
demonstrate that RoboPoint outperforms state-of-the-art VLMs (GPT-4o) and
visual prompting techniques (PIVOT) by 21.8% in the accuracy of predicting
spatial affordance and by 30.5% in the success rate of downstream tasks.
Project website: https://robo-point.github.io.
