  Emotion Recognition in Conversations (ERC) is crucial in developing
sympathetic human-machine interaction. In conversational videos, emotion can be
present in multiple modalities, i.e., audio, video, and transcript. However,
due to the inherent characteristics of these modalities, multi-modal ERC has
always been considered a challenging undertaking. Existing ERC research focuses
mainly on using text information in a discussion, ignoring the other two
modalities. We anticipate that emotion recognition accuracy can be improved by
employing a multi-modal approach. Thus, in this study, we propose a Multi-modal
Fusion Network (M2FNet) that extracts emotion-relevant features from visual,
audio, and text modality. It employs a multi-head attention-based fusion
mechanism to combine emotion-rich latent representations of the input data. We
introduce a new feature extractor to extract latent features from the audio and
visual modality. The proposed feature extractor is trained with a novel
adaptive margin-based triplet loss function to learn emotion-relevant features
from the audio and visual data. In the domain of ERC, the existing methods
perform well on one benchmark dataset but not on others. Our results show that
the proposed M2FNet architecture outperforms all other methods in terms of
weighted average F1 score on well-known MELD and IEMOCAP datasets and sets a
new state-of-the-art performance in ERC.
