  Large Vision-Language models (VLMs) have demonstrated strong reasoning
capabilities in tasks requiring a fine-grained understanding of literal images
and text, such as visual question-answering or visual entailment. However,
there has been little exploration of these models' capabilities when presented
with images and captions containing figurative phenomena such as metaphors or
humor, the meaning of which is often implicit. To close this gap, we propose a
new task and a high-quality dataset: Visual Figurative Language Understanding
with Textual Explanations (V-FLUTE). We frame the visual figurative language
understanding problem as an explainable visual entailment task, where the model
has to predict whether the image (premise) entails a claim (hypothesis) and
justify the predicted label with a textual explanation. Using a human-AI
collaboration framework, we build a high-quality dataset, V-FLUTE, that
contains 6,027 <image, claim, label, explanation> instances spanning five
diverse multimodal figurative phenomena: metaphors, similes, idioms, sarcasm,
and humor. The figurative phenomena can be present either in the image, the
caption, or both. We further conduct both automatic and human evaluations to
assess current VLMs' capabilities in understanding figurative phenomena.
