  In recent years, multimodal large language models (MLLMs) have shown
remarkable capabilities in tasks like visual question answering and common
sense reasoning, while visual perception models have made significant strides
in perception tasks, such as detection and segmentation. However, MLLMs mainly
focus on high-level image-text interpretations and struggle with fine-grained
visual understanding, and vision perception models usually suffer from
open-world distribution shifts due to their limited model capacity. To overcome
these challenges, we propose the Mutually Reinforced Multimodal Large Language
Model (MR-MLLM), a novel framework that synergistically enhances visual
perception and multimodal comprehension. First, a shared query fusion mechanism
is proposed to harmonize detailed visual inputs from vision models with the
linguistic depth of language models, enhancing multimodal comprehension and
vision perception synergistically. Second, we propose the perception-enhanced
cross-modal integration method, incorporating novel modalities from vision
perception outputs, like object detection bounding boxes, to capture subtle
visual elements, thus enriching the understanding of both visual and textual
data. In addition, an innovative perception-embedded prompt generation
mechanism is proposed to embed perceptual information into the language model's
prompts, aligning the responses contextually and perceptually for a more
accurate multimodal interpretation. Extensive experiments demonstrate MR-MLLM's
superior performance in various multimodal comprehension and vision perception
tasks, particularly those requiring corner case vision perception and
fine-grained language comprehension.
