  Recent breakthroughs in NLP research, such as the advent of Transformer
models have indisputably contributed to major advancements in several tasks.
However, few works research robustness and explainability issues of their
evaluation strategies. In this work, we examine the behavior of high-performing
pre-trained language models, focusing on the task of semantic similarity for
visual vocabularies. First, we address the need for explainable evaluation
metrics, necessary for understanding the conceptual quality of retrieved
instances. Our proposed metrics provide valuable insights in local and global
level, showcasing the inabilities of widely used approaches. Secondly,
adversarial interventions on salient query semantics expose vulnerabilities of
opaque metrics and highlight patterns in learned linguistic representations.
