  Multi-modal Multi-label Emotion Recognition (MMER) aims to identify various
human emotions from heterogeneous visual, audio and text modalities. Previous
methods mainly focus on projecting multiple modalities into a common latent
space and learning an identical representation for all labels, which neglects
the diversity of each modality and fails to capture richer semantic information
for each label from different perspectives. Besides, associated relationships
of modalities and labels have not been fully exploited. In this paper, we
propose versaTile multi-modAl learning for multI-labeL emOtion Recognition
(TAILOR), aiming to refine multi-modal representations and enhance
discriminative capacity of each label. Specifically, we design an adversarial
multi-modal refinement module to sufficiently explore the commonality among
different modalities and strengthen the diversity of each modality. To further
exploit label-modal dependence, we devise a BERT-like cross-modal encoder to
gradually fuse private and common modality representations in a granularity
descent way, as well as a label-guided decoder to adaptively generate a
tailored representation for each label with the guidance of label semantics. In
addition, we conduct experiments on the benchmark MMER dataset CMU-MOSEI in
both aligned and unaligned settings, which demonstrate the superiority of
TAILOR over the state-of-the-arts. Code is available at
https://github.com/kniter1/TAILOR.
