  Multimodal representation learning is a challenging task in which previous
work mostly focus on either uni-modality pre-training or cross-modality fusion.
In fact, we regard modeling multimodal representation as building a skyscraper,
where laying stable foundation and designing the main structure are equally
essential. The former is like encoding robust uni-modal representation while
the later is like integrating interactive information among different
modalities, both of which are critical to learning an effective multimodal
representation. Recently, contrastive learning has been successfully applied in
representation learning, which can be utilized as the pillar of the skyscraper
and benefit the model to extract the most important features contained in the
multimodal data. In this paper, we propose a novel framework named MultiModal
Contrastive Learning (MMCL) for multimodal representation to capture intra- and
inter-modality dynamics simultaneously. Specifically, we devise uni-modal
contrastive coding with an efficient uni-modal feature augmentation strategy to
filter inherent noise contained in acoustic and visual modality and acquire
more robust uni-modality representations. Besides, a pseudo siamese network is
presented to predict representation across different modalities, which
successfully captures cross-modal dynamics. Moreover, we design two contrastive
learning tasks, instance- and sentiment-based contrastive learning, to promote
the process of prediction and learn more interactive information related to
sentiment. Extensive experiments conducted on two public datasets demonstrate
that our method surpasses the state-of-the-art methods.
