  Among the various modes of communication in social media, the use of Internet
memes has emerged as a powerful means to convey political, psychological, and
socio-cultural opinions. Although memes are typically humorous in nature,
recent days have witnessed a proliferation of harmful memes targeted to abuse
various social entities. As most harmful memes are highly satirical and
abstruse without appropriate contexts, off-the-shelf multimodal models may not
be adequate to understand their underlying semantics. In this work, we propose
two novel problem formulations: detecting harmful memes and the social entities
that these harmful memes target. To this end, we present HarMeme, the first
benchmark dataset, containing 3,544 memes related to COVID-19. Each meme went
through a rigorous two-stage annotation process. In the first stage, we labeled
a meme as very harmful, partially harmful, or harmless; in the second stage, we
further annotated the type of target(s) that each harmful meme points to:
individual, organization, community, or society/general public/other. The
evaluation results using ten unimodal and multimodal models highlight the
importance of using multimodal signals for both tasks. We further discuss the
limitations of these models and we argue that more research is needed to
address these problems.
