  Alongside acoustic information, linguistic features based on speech
transcripts have been proven useful in Speech Emotion Recognition (SER).
However, due to the scarcity of emotion labelled data and the difficulty of
recognizing emotional speech, it is hard to obtain reliable linguistic features
and models in this research area. In this paper, we propose to fuse Automatic
Speech Recognition (ASR) outputs into the pipeline for joint training SER. The
relationship between ASR and SER is understudied, and it is unclear what and
how ASR features benefit SER. By examining various ASR outputs and fusion
methods, our experiments show that in joint ASR-SER training, incorporating
both ASR hidden and text output using a hierarchical co-attention fusion
approach improves the SER performance the most. On the IEMOCAP corpus, our
approach achieves 63.4% weighted accuracy, which is close to the baseline
results achieved by combining ground-truth transcripts. In addition, we also
present novel word error rate analysis on IEMOCAP and layer-difference analysis
of the Wav2vec 2.0 model to better understand the relationship between ASR and
SER.
