  Emotion recognition is a crucial task for human conversation understanding.
It becomes more challenging with the notion of multimodal data, e.g., language,
voice, and facial expressions. As a typical solution, the global- and the local
context information are exploited to predict the emotional label for every
single sentence, i.e., utterance, in the dialogue. Specifically, the global
representation could be captured via modeling of cross-modal interactions at
the conversation level. The local one is often inferred using the temporal
information of speakers or emotional shifts, which neglects vital factors at
the utterance level. Additionally, most existing approaches take fused features
of multiple modalities in an unified input without leveraging modality-specific
representations. Motivating from these problems, we propose the Relational
Temporal Graph Neural Network with Auxiliary Cross-Modality Interaction
(CORECT), an novel neural network framework that effectively captures
conversation-level cross-modality interactions and utterance-level temporal
dependencies with the modality-specific manner for conversation understanding.
Extensive experiments demonstrate the effectiveness of CORECT via its
state-of-the-art results on the IEMOCAP and CMU-MOSEI datasets for the
multimodal ERC task.
