  Human beings have rich ways of emotional expressions, including facial
action, voice, and natural languages. Due to the diversity and complexity of
different individuals, the emotions expressed by various modalities may be
semantically irrelevant. Directly fusing information from different modalities
may inevitably make the model subject to the noise from semantically irrelevant
modalities. To tackle this problem, we propose a multimodal relevance
estimation network to capture the relevant semantics among modalities in
multimodal emotions. Specifically, we take advantage of an attention mechanism
to reflect the semantic relevance weights of each modality. Moreover, we
propose a relevant semantic estimation loss to weakly supervise the semantics
of each modality. Furthermore, we make use of contrastive learning to optimize
the similarity of category-level modality-relevant semantics across different
modalities in feature space, thereby bridging the semantic gap between
heterogeneous modalities. In order to better reflect the emotional state in the
real interactive scenarios and perform the semantic relevance analysis, we
collect a single-label discrete multimodal emotion dataset named SDME, which
enables researchers to conduct multimodal semantic relevance research with
large category bias. Experiments on continuous and discrete emotion datasets
show that our model can effectively capture the relevant semantics, especially
for the large deviations in modal semantics. The code and SDME dataset will be
publicly available.
