  Spectrogram is commonly used as the input feature of deep neural networks to
learn the high(er)-level time-frequency pattern of speech signal for speech
emotion recognition (SER). \textcolor{black}{Generally, different emotions
correspond to specific energy activations both within frequency bands and time
frames on spectrogram, which indicates the frequency and time domains are both
essential to represent the emotion for SER. However, recent spectrogram-based
works mainly focus on modeling the long-term dependency in time domain, leading
to these methods encountering the following two issues: (1) neglecting to model
the emotion-related correlations within frequency domain during the
time-frequency joint learning; (2) ignoring to capture the specific frequency
bands associated with emotions.} To cope with the issues, we propose an
attentive time-frequency neural network (ATFNN) for SER, including a
time-frequency neural network (TFNN) and time-frequency attention.
Specifically, aiming at the first issue, we design a TFNN with a
frequency-domain encoder (F-Encoder) based on the Transformer encoder and a
time-domain encoder (T-Encoder) based on the Bidirectional Long Short-Term
Memory (Bi-LSTM). The F-Encoder and T-Encoder model the correlations within
frequency bands and time frames, respectively, and they are embedded into a
time-frequency joint learning strategy to obtain the time-frequency patterns
for speech emotions. Moreover, to handle the second issue, we also adopt
time-frequency attention with a frequency-attention network (F-Attention) and a
time-attention network (T-Attention) to focus on the emotion-related frequency
band ranges and time frame ranges, which can enhance the discriminability of
speech emotion features.
