  Sarcasm is a pervading linguistic phenomenon and highly challenging to
explain due to its subjectivity, lack of context and deeply-felt opinion. In
the multimodal setup, sarcasm is conveyed through the incongruity between the
text and visual entities. Although recent approaches deal with sarcasm as a
classification problem, it is unclear why an online post is identified as
sarcastic. Without proper explanation, end users may not be able to perceive
the underlying sense of irony. In this paper, we propose a novel problem --
Multimodal Sarcasm Explanation (MuSE) -- given a multimodal sarcastic post
containing an image and a caption, we aim to generate a natural language
explanation to reveal the intended sarcasm. To this end, we develop MORE, a new
dataset with explanation of 3510 sarcastic multimodal posts. Each explanation
is a natural language (English) sentence describing the hidden irony. We
benchmark MORE by employing a multimodal Transformer-based architecture. It
incorporates a cross-modal attention in the Transformer's encoder which attends
to the distinguishing features between the two modalities. Subsequently, a
BART-based auto-regressive decoder is used as the generator. Empirical results
demonstrate convincing results over various baselines (adopted for MuSE) across
five evaluation metrics. We also conduct human evaluation on predictions and
obtain Fleiss' Kappa score of 0.4 as a fair agreement among 25 evaluators.
