  Large Multimodal Model (LMM) is a hot research topic in the computer vision
area and has also demonstrated remarkable potential across multiple
disciplinary fields. A recent trend is to further extend and enhance the
perception capabilities of LMMs. The current methods follow the paradigm of
adapting the visual task outputs to the format of the language model, which is
the main component of a LMM. This adaptation leads to convenient development of
such LMMs with minimal modifications, however, it overlooks the intrinsic
characteristics of diverse visual tasks and hinders the learning of perception
capabilities. To address this issue, we propose a novel LMM architecture named
Lumen, a Large multimodal model with versatile vision-centric capability
enhancement. We decouple the LMM's learning of perception capabilities into
task-agnostic and task-specific stages. Lumen first promotes fine-grained
vision-language concept alignment, which is the fundamental capability for
various visual tasks. Thus the output of the task-agnostic stage is a shared
representation for all the tasks we address in this paper. Then the
task-specific decoding is carried out by flexibly routing the shared
representation to lightweight task decoders with negligible training efforts.
Comprehensive experimental results on a series of vision-centric and VQA
benchmarks indicate that our Lumen model not only achieves or surpasses the
performance of existing LMM-based approaches in a range of vision-centric tasks
while maintaining general visual understanding and instruction following
capabilities. The code will be released at https://github.com/SxJyJay/Lumen.
