  The exponential rise of online social media has enabled the creation,
distribution, and consumption of information at an unprecedented rate. However,
it has also led to the burgeoning of various forms of online abuse. Increasing
cases of online antisemitism have become one of the major concerns because of
its socio-political consequences. Unlike other major forms of online abuse like
racism, sexism, etc., online antisemitism has not been studied much from a
machine learning perspective. To the best of our knowledge, we present the
first work in the direction of automated multimodal detection of online
antisemitism. The task poses multiple challenges that include extracting
signals across multiple modalities, contextual references, and handling
multiple aspects of antisemitism. Unfortunately, there does not exist any
publicly available benchmark corpus for this critical task. Hence, we collect
and label two datasets with 3,102 and 3,509 social media posts from Twitter and
Gab respectively. Further, we present a multimodal deep learning system that
detects the presence of antisemitic content and its specific antisemitism
category using text and images from posts. We perform an extensive set of
experiments on the two datasets to evaluate the efficacy of the proposed
system. Finally, we also present a qualitative analysis of our study.
