  This paper presents novel benchmarks for evaluating vision-language models
(VLMs) in zero-shot recognition, focusing on granularity and specificity.
Although VLMs excel in tasks like image captioning, they face challenges in
open-world settings. Our benchmarks test VLMs' consistency in understanding
concepts across semantic granularity levels and their response to varying text
specificity. Findings show that VLMs favor moderately fine-grained concepts and
struggle with specificity, often misjudging texts that differ from their
training data. Extensive evaluations reveal limitations in current VLMs,
particularly in distinguishing between correct and subtly incorrect
descriptions. While fine-tuning offers some improvements, it doesn't fully
address these issues, highlighting the need for VLMs with enhanced
generalization capabilities for real-world applications. This study provides
insights into VLM limitations and suggests directions for developing more
robust models.
