  Image-based advertisements are complex multimodal stimuli that often contain
unusual visual elements and figurative language. Previous research on automatic
ad understanding has reported impressive zero-shot accuracy of contrastive
vision-and-language models (VLMs) on an ad-explanation retrieval task. Here, we
examine the original task setup and show that contrastive VLMs can solve it by
exploiting grounding heuristics. To control for this confound, we introduce
TRADE, a new evaluation test set with adversarial grounded explanations. While
these explanations look implausible to humans, we show that they "fool" four
different contrastive VLMs. Our findings highlight the need for an improved
operationalisation of automatic ad understanding that truly evaluates VLMs'
multimodal reasoning abilities. We make our code and TRADE available at
https://github.com/dmg-illc/trade .
