  Large vision-and-language models (VLMs) trained to match images with text on
large-scale datasets of image-text pairs have shown impressive generalization
ability on several vision and language tasks. Several recent works, however,
showed that these models lack fine-grained understanding, such as the ability
to count and recognize verbs, attributes, or relationships. The focus of this
work is to study the understanding of spatial relations. This has been tackled
previously using image-text matching (e.g., Visual Spatial Reasoning benchmark)
or visual question answering (e.g., GQA or VQAv2), both showing poor
performance and a large gap compared to human performance. In this work, we
show qualitatively (using explainability tools) and quantitatively (using
object detectors) that the poor object localization "grounding" ability of the
models is a contributing factor to the poor image-text matching performance. We
propose an alternative fine-grained, compositional approach for recognizing and
ranking spatial clauses that combines the evidence from grounding noun phrases
corresponding to objects and their locations to compute the final rank of the
spatial clause. We demonstrate the approach on representative VLMs (such as
LXMERT, GPV, and MDETR) and compare and highlight their abilities to reason
about spatial relationships.
