  The automatic recognition of emotion in speech can inform our understanding
of language, emotion, and the brain. It also has practical application to
human-machine interactive systems. This paper examines the recognition of
emotion in naturally occurring speech, where there are no constraints on what
is said or the emotions expressed. This task is more difficult than that using
data collected in scripted, experimentally controlled settings, and fewer
results are published. Our data come from couples in psychotherapy. Video and
audio recordings were made of three couples (A, B, C) over 18 hour-long therapy
sessions. This paper describes the method used to code the audio recordings for
the four emotions of Anger, Sadness, Joy and Tension, plus Neutral, also
covering our approach to managing the unbalanced samples that a naturally
occurring emotional speech dataset produces. Three groups of acoustic features
were used in our analysis: filter-bank, frequency, and voice-quality features.
The random forests model classified the features. Recognition rates are
reported for each individual, the result of the speaker-dependent models that
we built. In each case, the best recognition rates were achieved using the
filter-bank features alone. For Couple A, these rates were 90% for the female
and 87% for the male for the recognition of three emotions plus Neutral. For
Couple B, the rates were 84% for the female and 78% for the male for the
recognition of all four emotions plus Neutral. For Couple C, a rate of 88% was
achieved for the female for the recognition of the four emotions plus Neutral
and 95% for the male for three emotions plus Neutral. For pairwise recognition,
the rates ranged from 76% to 99% across the three couples. Our results show
that couple therapy is a rich context for the study of emotion in naturally
occurring speech.
