  In this work, we train fully convolutional networks to detect anger in
speech. Since training these deep architectures requires large amounts of data
and the size of emotion datasets is relatively small, we use transfer learning.
However, unlike previous approaches that use speech or emotion-based tasks for
the source model, we instead use SoundNet, a fully convolutional neural network
trained multimodally on a massive video dataset to classify audio, with
ground-truth labels provided by vision-based classifiers. As a result of
transfer learning from SoundNet, our trained anger detection model improves
performance and generalizes well on a variety of acted, elicited, and natural
emotional speech datasets. We also test the cross-lingual effectiveness of our
model by evaluating our English-trained model on Mandarin Chinese speech
emotion data. Furthermore, our proposed system has low latency suitable for
real-time applications, only requiring 1.2 seconds of audio to make a reliable
classification.
