  Emotion recognition is a challenging task due to limited availability of
in-the-wild labeled datasets. Self-supervised learning has shown improvements
on tasks with limited labeled datasets in domains like speech and natural
language. Models such as BERT learn to incorporate context in word embeddings,
which translates to improved performance in downstream tasks like question
answering. In this work, we extend self-supervised training to multi-modal
applications. We learn multi-modal representations using a transformer trained
on the masked language modeling task with audio, visual and text features. This
model is fine-tuned on the downstream task of emotion recognition. Our results
on the CMU-MOSEI dataset show that this pre-training technique can improve the
emotion recognition performance by up to 3% compared to the baseline.
