  The advent of large vision-language models (LVLMs) represents a remarkable
advance in the quest for artificial general intelligence. However, the model's
effectiveness in both specialized and general tasks warrants further
investigation. This paper endeavors to evaluate the competency of popular LVLMs
in specialized and general tasks, respectively, aiming to offer a comprehensive
understanding of these novel models. To gauge their effectiveness in
specialized tasks, we employ six challenging tasks in three different
application scenarios: natural, healthcare, and industrial. These six tasks
include salient/camouflaged/transparent object detection, as well as polyp
detection, skin lesion detection, and industrial anomaly detection. We examine
the performance of three recent open-source LVLMs, including MiniGPT-v2,
LLaVA-1.5, and Shikra, on both visual recognition and localization in these
tasks. Moreover, we conduct empirical investigations utilizing the
aforementioned LVLMs together with GPT-4V, assessing their multi-modal
understanding capabilities in general tasks including object counting, absurd
question answering, affordance reasoning, attribute recognition, and spatial
relation reasoning. Our investigations reveal that these LVLMs demonstrate
limited proficiency not only in specialized tasks but also in general tasks. We
delve deep into this inadequacy and uncover several potential factors,
including limited cognition in specialized tasks, object hallucination,
text-to-image interference, and decreased robustness in complex problems. We
hope that this study can provide useful insights for the future development of
LVLMs, helping researchers improve LVLMs for both general and specialized
applications.
