  Transformer has obtained promising results on cognitive speech signal
processing field, which is of interest in various applications ranging from
emotion to neurocognitive disorder analysis. However, most works treat speech
signal as a whole, leading to the neglect of the pronunciation structure that
is unique to speech and reflects the cognitive process. Meanwhile, Transformer
has heavy computational burden due to its full attention operation. In this
paper, a hierarchical efficient framework, called SpeechFormer, which considers
the structural characteristics of speech, is proposed and can be served as a
general-purpose backbone for cognitive speech signal processing. The proposed
SpeechFormer consists of frame, phoneme, word and utterance stages in
succession, each performing a neighboring attention according to the structural
pattern of speech with high computational efficiency. SpeechFormer is evaluated
on speech emotion recognition (IEMOCAP & MELD) and neurocognitive disorder
detection (Pitt & DAIC-WOZ) tasks, and the results show that SpeechFormer
outperforms the standard Transformer-based framework while greatly reducing the
computational cost. Furthermore, our SpeechFormer achieves comparable results
to the state-of-the-art approaches.
