  This work addresses the challenge of hate speech detection in Internet memes,
and attempts using visual information to automatically detect hate speech,
unlike any previous work of our knowledge. Memes are pixel-based multimedia
documents that contain photos or illustrations together with phrases which,
when combined, usually adopt a funny meaning. However, hate memes are also used
to spread hate through social networks, so their automatic detection would help
reduce their harmful societal impact. Our results indicate that the model can
learn to detect some of the memes, but that the task is far from being solved
with this simple architecture. While previous work focuses on linguistic hate
speech, our experiments indicate how the visual modality can be much more
informative for hate speech detection than the linguistic one in memes. In our
experiments, we built a dataset of 5,020 memes to train and evaluate a
multi-layer perceptron over the visual and language representations, whether
independently or fused. The source code and mode and models are available
https://github.com/imatge-upc/hate-speech-detection .
