  Multimodal Sentiment Analysis is an active area of research that leverages
multimodal signals for affective understanding of user-generated videos. The
predominant approach, addressing this task, has been to develop sophisticated
fusion techniques. However, the heterogeneous nature of the signals creates
distributional modality gaps that pose significant challenges. In this paper,
we aim to learn effective modality representations to aid the process of
fusion. We propose a novel framework, MISA, which projects each modality to two
distinct subspaces. The first subspace is modality-invariant, where the
representations across modalities learn their commonalities and reduce the
modality gap. The second subspace is modality-specific, which is private to
each modality and captures their characteristic features. These representations
provide a holistic view of the multimodal data, which is used for fusion that
leads to task predictions. Our experiments on popular sentiment analysis
benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art
models. We also consider the task of Multimodal Humor Detection and experiment
on the recently proposed UR_FUNNY dataset. Here too, our model fares better
than strong baselines, establishing MISA as a useful multimodal framework.
