  Although automatic emotion recognition (AER) has recently drawn significant
research interest, most current AER studies use manually segmented utterances,
which are usually unavailable for dialogue systems. This paper proposes
integrating AER with automatic speech recognition (ASR) and speaker diarisation
(SD) in a jointly-trained system. Distinct output layers are built for four
sub-tasks including AER, ASR, voice activity detection and speaker
classification based on a shared encoder. Taking the audio of a conversation as
input, the integrated system finds all speech segments and transcribes the
corresponding emotion classes, word sequences, and speaker identities. Two
metrics are proposed to evaluate AER performance with automatic segmentation
based on time-weighted emotion and speaker classification errors. Results on
the IEMOCAP dataset show that the proposed system consistently outperforms two
baselines with separately trained single-task systems on AER, ASR and SD.
