  The rise of Multimodal Large Language Models (MLLMs), renowned for their
advanced instruction-following and reasoning capabilities, has significantly
propelled the field of visual reasoning. However, due to limitations in their
image tokenization processes, most MLLMs struggle to capture fine details of
text and objects in images, especially in high-resolution samples. To overcome
this limitation, we introduce P2G, a novel framework for plug-and-play
grounding in MLLMs. P2G utilizes the tool-usage potential of MLLMs to employ
expert agents for on-the-fly grounding of reasoning into critical visual and
textual elements in images, thereby enabling deliberate reasoning through
multimodal prompting. Additionally, we develop P2GB, a benchmark designed to
evaluate MLLMs' proficiency in understanding inter-object relationships and
textual content in challenging high-resolution images. Extensive experiments on
visual reasoning tasks demonstrate the superiority of P2G, achieving
performance comparable to GPT-4V on P2GB with a 7B backbone. Our work
underscores the potential of grounding reasoning with external agents in MLLMs,
presenting a promising alternative to mere model scaling.
