  The way that humans encode their emotion into speech signals is complex. For
instance, an angry man may increase his pitch and speaking rate, and use
impolite words. In this paper, we present a preliminary study on various
emotional factors and investigate how each of them impacts modern emotion
recognition systems. The key tool of our study is the SpeechFlow model
presented recently, by which we are able to decompose speech signals into
separate information factors (content, pitch, rhythm). Based on this
decomposition, we carefully studied the performance of each information
component and their combinations. We conducted the study on three different
speech emotion corpora and chose an attention-based convolutional RNN as the
emotion classifier. Our results show that rhythm is the most important
component for emotional expression. Moreover, the cross-corpus results are very
bad (even worse than guess), demonstrating that the present speech emotion
recognition model is rather weak. Interestingly, by removing one or several
unimportant components, the cross-corpus results can be improved. This
demonstrates the potential of the decomposition approach towards a
generalizable emotion recognition.
