  Internet memes have become a dominant method of communication; at the same
time, however, they are also increasingly being used to advocate extremism and
foster derogatory beliefs. Nonetheless, we do not have a firm understanding as
to which perceptual aspects of memes cause this phenomenon. In this work, we
assess the efficacy of current state-of-the-art multimodal machine learning
models toward hateful meme detection, and in particular with respect to their
generalizability across platforms. We use two benchmark datasets comprising
12,140 and 10,567 images from 4chan's "Politically Incorrect" board (/pol/) and
Facebook's Hateful Memes Challenge dataset to train the competition's
top-ranking machine learning models for the discovery of the most prominent
features that distinguish viral hateful memes from benign ones. We conduct
three experiments to determine the importance of multimodality on
classification performance, the influential capacity of fringe Web communities
on mainstream social platforms and vice versa, and the models' learning
transferability on 4chan memes. Our experiments show that memes' image
characteristics provide a greater wealth of information than its textual
content. We also find that current systems developed for online detection of
hate speech in memes necessitate further concentration on its visual elements
to improve their interpretation of underlying cultural connotations, implying
that multimodal models fail to adequately grasp the intricacies of hate speech
in memes and generalize across social media platforms.
