  Most current audio-visual emotion recognition models lack the flexibility
needed for deployment in practical applications. We envision a multimodal
system that works even when only one modality is available and can be
implemented interchangeably for either predicting emotional attributes or
recognizing categorical emotions. Achieving such flexibility in a multimodal
emotion recognition system is difficult due to the inherent challenges in
accurately interpreting and integrating varied data sources. It is also a
challenge to robustly handle missing or partial information while allowing
direct switch between regression or classification tasks. This study proposes a
versatile audio-visual learning (VAVL) framework for handling unimodal and
multimodal systems for emotion regression or emotion classification tasks. We
implement an audio-visual framework that can be trained even when audio and
visual paired data is not available for part of the training set (i.e., audio
only or only video is present). We achieve this effective representation
learning with audio-visual shared layers, residual connections over shared
layers, and a unimodal reconstruction task. Our experimental results reveal
that our architecture significantly outperforms strong baselines on the
CREMA-D, MSP-IMPROV, and CMU-MOSEI corpora. Notably, VAVL attains a new
state-of-the-art performance in the emotional attribute prediction task on the
MSP-IMPROV corpus.
