  Sarcasm is a type of irony, characterized by an inherent mismatch between the
literal interpretation and the intended connotation. Though sarcasm detection
in text has been extensively studied, there are situations in which textual
input alone might be insufficient to perceive sarcasm. The inclusion of
additional contextual cues, such as images, is essential to recognize sarcasm
in social media data effectively. This study presents a novel framework for
multimodal sarcasm detection that can process input triplets. Two components of
these triplets comprise the input text and its associated image, as provided in
the datasets. Additionally, a supplementary modality is introduced in the form
of descriptive image captions. The motivation behind incorporating this visual
semantic representation is to more accurately capture the discrepancies between
the textual and visual content, which are fundamental to the sarcasm detection
task. The primary contributions of this study are: (1) a robust textual feature
extraction branch that utilizes a cross-lingual language model; (2) a visual
feature extraction branch that incorporates a self-regulated residual ConvNet
integrated with a lightweight spatially aware attention module; (3) an
additional modality in the form of image captions generated using an
encoder-decoder architecture capable of reading text embedded in images; (4)
distinct attention modules to effectively identify the incongruities between
the text and two levels of image representations; (5) multi-level cross-domain
semantic incongruity representation achieved through feature fusion. Compared
with cutting-edge baselines, the proposed model achieves the best accuracy of
92.89% and 64.48%, respectively, on the Twitter multimodal sarcasm and
MultiBully datasets.
