  Humans are skilled in reading the interlocutor's emotion from multimodal
signals, including spoken words, simultaneous speech, and facial expressions.
It is still a challenge to effectively decode emotions from the complex
interactions of multimodal signals. In this paper, we design three kinds of
multimodal latent representations to refine the emotion analysis process and
capture complex multimodal interactions from different views, including a
intact three-modal integrating representation, a modality-shared
representation, and three modality-individual representations. Then, a
modality-semantic hierarchical fusion is proposed to reasonably incorporate
these representations into a comprehensive interaction representation. The
experimental results demonstrate that our EffMulti outperforms the
state-of-the-art methods. The compelling performance benefits from its
well-designed framework with ease of implementation, lower computing
complexity, and less trainable parameters.
