  Existing self-supervised learning strategies are constrained to either a
limited set of objectives or generic downstream tasks that predominantly target
uni-modal applications. This has isolated progress for imperative multi-modal
applications that are diverse in terms of complexity and domain-affinity, such
as meme analysis. Here, we introduce two self-supervised pre-training methods,
namely Ext-PIE-Net and MM-SimCLR that (i) employ off-the-shelf multi-modal
hate-speech data during pre-training and (ii) perform self-supervised learning
by incorporating multiple specialized pretext tasks, effectively catering to
the required complex multi-modal representation learning for meme analysis. We
experiment with different self-supervision strategies, including potential
variants that could help learn rich cross-modality representations and evaluate
using popular linear probing on the Hateful Memes task. The proposed solutions
strongly compete with the fully supervised baseline via label-efficient
training while distinctly outperforming them on all three tasks of the Memotion
challenge with 0.18%, 23.64%, and 0.93% performance gain, respectively.
Further, we demonstrate the generalizability of the proposed solutions by
reporting competitive performance on the HarMeme task. Finally, we empirically
establish the quality of the learned representations by analyzing task-specific
learning, using fewer labeled training samples, and arguing that the complexity
of the self-supervision strategy and downstream task at hand are correlated.
Our efforts highlight the requirement of better multi-modal self-supervision
methods involving specialized pretext tasks for efficient fine-tuning and
generalizable performance.
