  Despite their remarkable successes, state-of-the-art language models face
challenges in grasping certain important semantic details. This paper
introduces the VISLA (Variance and Invariance to Semantic and Lexical
Alterations) benchmark, designed to evaluate the semantic and lexical
understanding of language models. VISLA presents a 3-way semantic
(in)equivalence task with a triplet of sentences associated with an image, to
evaluate both vision-language models (VLMs) and unimodal language models
(ULMs). An evaluation involving 34 VLMs and 20 ULMs reveals surprising
difficulties in distinguishing between lexical and semantic variations. Spatial
semantics encoded by language models also appear to be highly sensitive to
lexical information. Notably, text encoders of VLMs demonstrate greater
sensitivity to semantic and lexical variations than unimodal text encoders. Our
contributions include the unification of image-to-text and text-to-text
retrieval tasks, an off-the-shelf evaluation without fine-tuning, and assessing
LMs' semantic (in)variance in the presence of lexical alterations. The results
highlight strengths and weaknesses across diverse vision and unimodal language
models, contributing to a deeper understanding of their capabilities. % VISLA
enables a rigorous evaluation, shedding light on language models' capabilities
in handling semantic and lexical nuances. Data and code will be made available
at https://github.com/Sri-Harsha/visla_benchmark.
