  Internet memes have emerged as an increasingly popular means of communication
on the Web. Although typically intended to elicit humour, they have been
increasingly used to spread hatred, trolling, and cyberbullying, as well as to
target specific individuals, communities, or society on political,
socio-cultural, and psychological grounds. While previous work has focused on
detecting harmful, hateful, and offensive memes, identifying whom they attack
remains a challenging and underexplored area. Here we aim to bridge this gap.
In particular, we create a dataset where we annotate each meme with its
victim(s) such as the name of the targeted person(s), organization(s), and
community(ies). We then propose DISARM (Detecting vIctimS targeted by hARmful
Memes), a framework that uses named entity recognition and person
identification to detect all entities a meme is referring to, and then,
incorporates a novel contextualized multimodal deep neural network to classify
whether the meme intends to harm these entities. We perform several systematic
experiments on three test setups, corresponding to entities that are (a) all
seen while training, (b) not seen as a harmful target on training, and (c) not
seen at all on training. The evaluation results show that DISARM significantly
outperforms ten unimodal and multimodal systems. Finally, we show that DISARM
is interpretable and comparatively more generalizable and that it can reduce
the relative error rate for harmful target identification by up to 9 points
absolute over several strong multimodal rivals.
