  Large Vision and Language Models have enabled significant advances in fully
supervised and zero-shot visual tasks. These large architectures serve as the
baseline to what is currently known as Instruction Tuning Large Vision and
Language models (IT-LVLMs). IT-LVLMs are general-purpose multi-modal assistants
whose responses are modulated by natural language instructions and visual data.
Despite this versatility, IT-LVLM effectiveness in fundamental computer vision
problems remains unclear, primarily due to the absence of a standardized
evaluation benchmark. This paper introduces a Multi-modal Evaluation Benchmark
named MERLIM, a scalable test-bed to assess the capabilities of IT-LVLMs on
fundamental computer vision tasks. MERLIM contains over 300K image-question
pairs and has a strong focus on detecting cross-modal "hallucination" events in
IT-LVLMs. Our results bring important insights on the performance of
state-of-the-art IT-LVMLs including limitations at identifying fine-grained
visual concepts, object hallucinations across tasks, and biases towards the
language query. Our findings also suggest that these models have weak visual
grounding, but manage to make adequate guesses from global visual patterns or
language biases contained in the LLM component.
