  Vision and language models (VLMs) such as CLIP have showcased remarkable
zero-shot recognition abilities yet face challenges in visio-linguistic
compositionality, particularly in linguistic comprehension and fine-grained
image-text alignment. This paper explores the intricate relationship between
compositionality and recognition -- two pivotal aspects of VLM capability. We
conduct a comprehensive evaluation of existing VLMs, covering both pre-training
approaches aimed at recognition and the fine-tuning methods designed to improve
compositionality. Our evaluation employs 12 benchmarks for compositionality,
along with 21 zero-shot classification and two retrieval benchmarks for
recognition. In our analysis from 274 CLIP model checkpoints, we reveal
patterns and trade-offs that emerge between compositional understanding and
recognition accuracy. Ultimately, this necessitates strategic efforts towards
developing models that improve both capabilities, as well as the meticulous
formulation of benchmarks for compositionality. We open our evaluation
framework at https://github.com/ytaek-oh/vl_compo.
