  Online content is filled with logos, from ads and social media posts to
website branding and product placements. Consequently, these logos are
prevalent in the extensive web-scraped datasets used to pretrain
Vision-Language Models, which are used for a wide array of tasks (content
moderation, object classification). While these models have been shown to learn
harmful correlations in various tasks, whether these correlations include logos
remains understudied. Understanding this is especially important due to logos
often being used by public-facing entities like brands and government agencies.
To that end, we develop SLANT: A Spurious Logo ANalysis Toolkit. Our key
finding is that some logos indeed lead to spurious incorrect predictions, for
example, adding the Adidas logo to a photo of a person causes a model classify
the person as greedy. SLANT contains a semi-automatic mechanism for mining such
"spurious" logos. The mechanism consists of a comprehensive logo bank,
CC12M-LogoBank, and an algorithm that searches the bank for logos that VLMs
spuriously correlate with a user-provided downstream recognition target. We
uncover various seemingly harmless logos that VL models correlate 1) with
negative human adjectives 2) with the concept of `harmlessness'; causing models
to misclassify harmful online content as harmless, and 3) with user-provided
object concepts; causing lower recognition accuracy on ImageNet zero-shot
classification. Furthermore, SLANT's logos can be seen as effective attacks
against foundational models; an attacker could place a spurious logo on harmful
content, causing the model to misclassify it as harmless. This threat is
alarming considering the simplicity of logo attacks, increasing the attack
surface of VL models. As a defense, we include in our Toolkit two effective
mitigation strategies that seamlessly integrate with zero-shot inference of
foundation models.
