  Recent advances in Large Language Models (LLMs) have stimulated a surge of
research aimed at extending their applications to the visual domain. While
these models exhibit promise in generating abstract image captions and
facilitating natural conversations, their performance on text-rich images still
requires improvement. In this paper, we introduce Contrastive Reading Model
(Cream), a novel neural architecture designed to enhance the language-image
understanding capability of LLMs by capturing intricate details that are often
overlooked in existing methods. Cream combines vision and auxiliary encoders,
fortified by a contrastive feature alignment technique, to achieve a more
effective comprehension of language information in visually situated contexts
within the images. Our approach bridges the gap between vision and language
understanding, paving the way for the development of more sophisticated
Document Intelligence Assistants. Through rigorous evaluations across diverse
visually-situated language understanding tasks that demand reasoning
capabilities, we demonstrate the compelling performance of Cream, positioning
it as a prominent model in the field of visual document understanding. We
provide our codebase and newly-generated datasets at
https://github.com/naver-ai/cream .
