  Vision-Language Models (VLMs) have shown remarkable capabilities in a large
number of downstream tasks. Nonetheless, compositional image understanding
remains a rather difficult task due to the object bias present in training
data. In this work, we investigate the reasons for such a lack of capability by
performing an extensive bench-marking of compositional understanding in VLMs.
We compare contrastive models with generative ones and analyze their
differences in architecture, pre-training data, and training tasks and losses.
Furthermore, we leverage In-Context Learning (ICL) as a way to improve the
ability of VLMs to perform more complex reasoning and understanding given an
image. Our extensive experiments demonstrate that our proposed approach
outperforms baseline models across multiple compositional understanding
datasets.
