  With the continuous development of deep learning (DL), the task of multimodal
dialogue emotion recognition (MDER) has recently received extensive research
attention, which is also an essential branch of DL. The MDER aims to identify
the emotional information contained in different modalities, e.g., text, video,
and audio, in different dialogue scenes. However, existing research has focused
on modeling contextual semantic information and dialogue relations between
speakers while ignoring the impact of event relations on emotion. To tackle the
above issues, we propose a novel Dialogue and Event Relation-Aware Graph
Convolutional Neural Network for Multimodal Emotion Recognition (DER-GCN)
method. It models dialogue relations between speakers and captures latent event
relations information. Specifically, we construct a weighted multi-relationship
graph to simultaneously capture the dependencies between speakers and event
relations in a dialogue. Moreover, we also introduce a Self-Supervised Masked
Graph Autoencoder (SMGAE) to improve the fusion representation ability of
features and structures. Next, we design a new Multiple Information Transformer
(MIT) to capture the correlation between different relations, which can provide
a better fuse of the multivariate information between relations. Finally, we
propose a loss optimization strategy based on contrastive learning to enhance
the representation learning ability of minority class features. We conduct
extensive experiments on the IEMOCAP and MELD benchmark datasets, which verify
the effectiveness of the DER-GCN model. The results demonstrate that our model
significantly improves both the average accuracy and the f1 value of emotion
recognition.
