  Emotion recognition from speech is one of the key steps towards emotional
intelligence in advanced human-machine interaction. Identifying emotions in
human speech requires learning features that are robust and discriminative
across diverse domains that differ in terms of language, spontaneity of speech,
recording conditions, and types of emotions. This corresponds to a learning
scenario in which the joint distributions of features and labels may change
substantially across domains. In this paper, we propose a deep architecture
that jointly exploits a convolutional network for extracting domain-shared
features and a long short-term memory network for classifying emotions using
domain-specific features. We use transferable features to enable model
adaptation from multiple source domains, given the sparseness of speech emotion
data and the fact that target domains are short of labeled data. A
comprehensive cross-corpora experiment with diverse speech emotion domains
reveals that transferable features provide gains ranging from 4.3% to 18.4% in
speech emotion recognition. We evaluate several domain adaptation approaches,
and we perform an ablation study to understand which source domains add the
most to the overall recognition effectiveness for a given target domain.
