  There is a rapidly growing need for multimodal content moderation (CM) as
more and more content on social media is multimodal in nature. Existing
unimodal CM systems may fail to catch harmful content that crosses modalities
(e.g., memes or videos), which may lead to severe consequences. In this paper,
we present a novel CM model, Asymmetric Mixed-Modal Moderation (AM3), to target
multimodal and unimodal CM tasks. Specifically, to address the asymmetry in
semantics between vision and language, AM3 has a novel asymmetric fusion
architecture that is designed to not only fuse the common knowledge in both
modalities but also to exploit the unique information in each modality. Unlike
previous works that focus on representing the two modalities into a similar
feature space while overlooking the intrinsic difference between the
information conveyed in multimodality and in unimodality (asymmetry in
modalities), we propose a novel cross-modality contrastive loss to learn the
unique knowledge that only appears in multimodality. This is critical as some
harmful intent may only be conveyed through the intersection of both
modalities. With extensive experiments, we show that AM3 outperforms all
existing state-of-the-art methods on both multimodal and unimodal CM
benchmarks.
