  Smaller-scale Vision-Langauge Models (VLMs) often claim to perform on par
with larger models in general-domain visual grounding and question-answering
benchmarks while offering advantages in computational efficiency and storage.
However, their ability to handle rare objects, which fall into the long tail of
data distributions, is less understood. To rigorously evaluate this aspect, we
introduce the "Uncontextualized Uncommon Objects" (UOUO) benchmark. This
benchmark focuses on systematically testing VLMs with both large and small
parameter counts on rare and specialized objects. Our comprehensive analysis
reveals that while smaller VLMs maintain competitive performance on common
datasets, they significantly underperform on tasks involving uncommon objects.
We also propose an advanced, scalable pipeline for data collection and
cleaning, ensuring the UOUO benchmark provides high-quality, challenging
instances. These findings highlight the need to consider long-tail
distributions when assessing the true capabilities of VLMs.
