  Training a Multimodal Large Language Model (MLLM) from scratch, like GPT-4,
is resource-intensive. Regarding Large Language Models (LLMs) as the core
processor for multimodal information, our paper introduces LMEye, a human-like
eye with a play-and-plug interactive perception network, designed to enable
dynamic interaction between LLMs and external vision information. Previous
methods incorporate visual information into LLMs with a simple visual mapping
network or Q-former from BLIP-2. Such networks project the image feature once
yet do not consider the interaction between the image and the human input
query. Hence, the obtained visual information without being connected to human
intention may be inadequate for LLMs to generate intention-following responses,
which we refer to as static visual information. LMEye addresses this issue by
allowing the LLM to request the desired visual information aligned with various
human instructions, which we term as the dynamic visual information
interaction. Specifically, LMEye consists of a simple visual mapping network to
provide the basic perception of an image for LLMs. It also contains additional
modules responsible for acquiring requests from LLMs, performing request-based
visual information interaction, and transmitting the resulting interacted
visual information to LLMs, respectively. In this way, LLMs act to understand
the human query, deliver the corresponding request to the request-based visual
information interaction module, and generate the response based on the
interleaved multimodal information. We evaluate LMEye through extensive
experiments on some multimodal benchmarks, demonstrating that it significantly
improves the zero-shot performance on various multimodal tasks compared to
previous methods, with less parameters.
