  This paper delves into the formidable challenge of cross-domain
generalization in multimodal hate meme detection, presenting compelling
findings. We provide enough pieces of evidence supporting the hypothesis that
only the textual component of hateful memes enables the existing multimodal
classifier to generalize across different domains, while the image component
proves highly sensitive to a specific training dataset. The evidence includes
demonstrations showing that hate-text classifiers perform similarly to
hate-meme classifiers in a zero-shot setting. Simultaneously, the introduction
of captions generated from images of memes to the hate-meme classifier worsens
performance by an average F1 of 0.02. Through blackbox explanations, we
identify a substantial contribution of the text modality (average of 83%),
which diminishes with the introduction of meme's image captions (52%).
Additionally, our evaluation on a newly created confounder dataset reveals
higher performance on text confounders as compared to image confounders with an
average $\Delta$F1 of 0.18.
