  In this paper, we propose to improve emotion recognition by combining
acoustic information and conversation transcripts. On the one hand, an LSTM
network was used to detect emotion from acoustic features like f0, shimmer,
jitter, MFCC, etc. On the other hand, a multi-resolution CNN was used to detect
emotion from word sequences. This CNN consists of several parallel convolutions
with different kernel sizes to exploit contextual information at different
levels. A temporal pooling layer aggregates the hidden representations of
different words into a unique sequence level embedding, from which we computed
the emotion posteriors. We optimized a weighted sum of classification and
verification losses. The verification loss tries to bring embeddings from the
same emotions closer while separating embeddings from different emotions. We
also compared our CNN with state-of-the-art text-based hand-crafted features
(e-vector). We evaluated our approach on the USC-IEMOCAP dataset as well as the
dataset consisting of US English telephone speech. In the former, we used
human-annotated transcripts while in the latter, we used ASR transcripts. The
results showed fusing audio and transcript information improved unweighted
accuracy by relative 24% for IEMOCAP and relative 3.4% for the telephone data
compared to a single acoustic system.
