  The emergence of multimodal large models (MLMs) has significantly advanced
the field of visual understanding, offering remarkable capabilities in the
realm of visual question answering (VQA). Yet, the true challenge lies in the
domain of knowledge-intensive VQA tasks, which necessitate not just recognition
of visual elements, but also a deep comprehension of the visual information in
conjunction with a vast repository of learned knowledge. To uncover such
capabilities of MLMs, particularly the newly introduced GPT-4V, we provide an
in-depth evaluation from three perspectives: 1) Commonsense Knowledge, which
assesses how well models can understand visual cues and connect to general
knowledge; 2) Fine-grained World Knowledge, which tests the model's skill in
reasoning out specific knowledge from images, showcasing their proficiency
across various specialized fields; 3) Comprehensive Knowledge with
Decision-making Rationales, which examines model's capability to provide
logical explanations for its inference, facilitating a deeper analysis from the
interpretability perspective. Extensive experiments indicate that GPT-4V
achieves SOTA performance on above three tasks. Interestingly, we find that: a)
GPT-4V demonstrates enhanced reasoning and explanation when using composite
images as few-shot; b) GPT-4V produces severe hallucinations when dealing with
world knowledge, highlighting the future need for advancements in this research
direction.
