  In this paper, we introduce a pretrained audio-visual Transformer trained on
more than 500k utterances from nearly 4000 celebrities from the VoxCeleb2
dataset for human behavior understanding. The model aims to capture and extract
useful information from the interactions between human facial and auditory
behaviors, with application in emotion recognition. We evaluate the model
performance on two datasets, namely CREMAD-D (emotion classification) and
MSP-IMPROV (continuous emotion regression). Experimental results show that
fine-tuning the pre-trained model helps improving emotion classification
accuracy by 5-7% and Concordance Correlation Coefficients (CCC) in continuous
emotion recognition by 0.03-0.09 compared to the same model trained from
scratch. We also demonstrate the robustness of finetuning the pre-trained model
in a low-resource setting. With only 10% of the original training set provided,
fine-tuning the pre-trained model can lead to at least 10% better emotion
recognition accuracy and a CCC score improvement by at least 0.1 for continuous
emotion recognition.
