  Vision and Language Models (VLMs) continue to demonstrate remarkable
zero-shot (ZS) performance across various tasks. However, many probing studies
have revealed that even the best-performing VLMs struggle to capture aspects of
compositional scene understanding, lacking the ability to properly ground and
localize linguistic phrases in images. Recent VLM advancements include scaling
up both model and dataset sizes, additional training objectives and levels of
supervision, and variations in the model architectures. To characterize the
grounding ability of VLMs, such as phrase grounding, referring expressions
comprehension, and relationship understanding, Pointing Game has been used as
an evaluation metric for datasets with bounding box annotations. In this paper,
we introduce a novel suite of quantitative metrics that utilize GradCAM
activations to rigorously evaluate the grounding capabilities of pre-trained
VLMs like CLIP, BLIP, and ALBEF. These metrics offer an explainable and
quantifiable approach for a more detailed comparison of the zero-shot
capabilities of VLMs and enable measuring models' grounding uncertainty. This
characterization reveals interesting tradeoffs between the size of the model,
the dataset size, and their performance.
