  Multimodal emotion recognition in conversation (ERC) has garnered growing
attention from research communities in various fields. In this paper, we
propose a Cross-modal Fusion Network with Emotion-Shift Awareness (CFN-ESA) for
ERC. Extant approaches employ each modality equally without distinguishing the
amount of emotional information in these modalities, rendering it hard to
adequately extract complementary information from multimodal data. To cope with
this problem, in CFN-ESA, we treat textual modality as the primary source of
emotional information, while visual and acoustic modalities are taken as the
secondary sources. Besides, most multimodal ERC models ignore emotion-shift
information and overfocus on contextual information, leading to the failure of
emotion recognition under emotion-shift scenario. We elaborate an emotion-shift
module to address this challenge. CFN-ESA mainly consists of unimodal encoder
(RUME), cross-modal encoder (ACME), and emotion-shift module (LESM). RUME is
applied to extract conversation-level contextual emotional cues while pulling
together data distributions between modalities; ACME is utilized to perform
multimodal interaction centered on textual modality; LESM is used to model
emotion shift and capture emotion-shift information, thereby guiding the
learning of the main task. Experimental results demonstrate that CFN-ESA can
effectively promote performance for ERC and remarkably outperform
state-of-the-art models.
