  The exponential growth of social media has profoundly transformed how
information is created, disseminated, and absorbed, exceeding any precedent in
the digital age. Regrettably, this explosion has also spawned a significant
increase in the online abuse of memes. Evaluating the negative impact of memes
is notably challenging, owing to their often subtle and implicit meanings,
which are not directly conveyed through the overt text and imagery. In light of
this, large multimodal models (LMMs) have emerged as a focal point of interest
due to their remarkable capabilities in handling diverse multimodal tasks. In
response to this development, our paper aims to thoroughly examine the capacity
of various LMMs (e.g., GPT-4V) to discern and respond to the nuanced aspects of
social abuse manifested in memes. We introduce the comprehensive meme
benchmark, GOAT-Bench, comprising over 6K varied memes encapsulating themes
such as implicit hate speech, sexism, and cyberbullying, etc. Utilizing
GOAT-Bench, we delve into the ability of LMMs to accurately assess hatefulness,
misogyny, offensiveness, sarcasm, and harmful content. Our extensive
experiments across a range of LMMs reveal that current models still exhibit a
deficiency in safety awareness, showing insensitivity to various forms of
implicit abuse. We posit that this shortfall represents a critical impediment
to the realization of safe artificial intelligence. The GOAT-Bench and
accompanying resources are publicly accessible at https://goatlmm.github.io/,
contributing to ongoing research in this vital field.
