  Multimodal emotion recognition has attracted much attention recently. Fusing
multiple modalities effectively with limited labeled data is a challenging
task. Considering the success of pre-trained model and fine-grained nature of
emotion expression, it is reasonable to take these two aspects into
consideration. Unlike previous methods that mainly focus on one aspect, we
introduce a novel multi-granularity framework, which combines fine-grained
representation with pre-trained utterance-level representation. Inspired by
Transformer TTS, we propose a multilevel transformer model to perform
fine-grained multimodal emotion recognition. Specifically, we explore different
methods to incorporate phoneme-level embedding with word-level embedding. To
perform multi-granularity learning, we simply combine multilevel transformer
model with Albert. Extensive experimental results show that both our multilevel
transformer model and multi-granularity model outperform previous
state-of-the-art approaches on IEMOCAP dataset with text transcripts and speech
signal.
