  The limited availability of labeled data is a major challenge in audiovisual
speech emotion recognition (SER). Self-supervised learning approaches have
recently been proposed to mitigate the need for labeled data in various
applications. This paper proposes the VQ-MAE-AV model, a vector quantized
masked autoencoder (MAE) designed for audiovisual speech self-supervised
representation learning and applied to SER. Unlike previous approaches, the
proposed method employs a self-supervised paradigm based on discrete audio and
visual speech representations learned by vector quantized variational
autoencoders. A multimodal MAE with self- or cross-attention mechanisms is
proposed to fuse the audio and visual speech modalities and to learn local and
global representations of the audiovisual speech sequence, which are then used
for an SER downstream task. Experimental results show that the proposed
approach, which is pre-trained on the VoxCeleb2 database and fine-tuned on
standard emotional audiovisual speech datasets, outperforms the
state-of-the-art audiovisual SER methods. Extensive ablation experiments are
also provided to assess the contribution of the different model components.
