  Recently, self-supervised pre-training has shown significant improvements in
many areas of machine learning, including speech and NLP. We propose using
large self-supervised pre-trained models for both audio and text modality with
cross-modality attention for multimodal emotion recognition. We use Wav2Vec2.0
[1] as an audio encoder base for robust speech features extraction and the BERT
model [2] as a text encoder base for better contextual representation of text.
These high capacity models trained on large amounts of unlabeled data contain
rich feature representations and improve the downstream task's performance. We
use the cross-modal attention [3] mechanism to learn alignment between audio
and text representations from self-supervised models. Cross-modal attention
also helps in extracting interactive information between audio and text
features. We obtain utterance-level feature representation from frame-level
features using statistics pooling for both audio and text modality and combine
them using the early fusion technique. Our experiments show that the proposed
approach obtains a 1.88% absolute improvement in accuracy compared to the
previous state-of-the-art method [3] on the IEMOCAP dataset [35]. We also
conduct unimodal experiments for both audio and text modalities and compare
them with previous best methods.
