  The integration of Multimodal Large Language Models (MLLMs) with robotic
systems has significantly enhanced the ability of robots to interpret and act
upon natural language instructions. Despite these advancements, conventional
MLLMs are typically trained on generic image-text pairs, lacking essential
robotics knowledge such as affordances and physical knowledge, which hampers
their efficacy in manipulation tasks. To bridge this gap, we introduce
ManipVQA, a novel framework designed to endow MLLMs with Manipulation-centric
knowledge through a Visual Question-Answering format. This approach not only
encompasses tool detection and affordance recognition but also extends to a
comprehensive understanding of physical concepts. Our approach starts with
collecting a varied set of images displaying interactive objects, which
presents a broad range of challenges in tool object detection, affordance, and
physical concept predictions. To seamlessly integrate this robotic-specific
knowledge with the inherent vision-reasoning capabilities of MLLMs, we adopt a
unified VQA format and devise a fine-tuning strategy that preserves the
original vision-reasoning abilities while incorporating the new robotic
insights. Empirical evaluations conducted in robotic simulators and across
various vision task benchmarks demonstrate the robust performance of ManipVQA.
Code and dataset will be made publicly available at
https://github.com/SiyuanHuang95/ManipVQA.
