  Self-attention (SA), which encodes vector sequences according to their
pairwise similarity, is widely used in speech recognition due to its strong
context modeling ability. However, when applied to long sequence data, its
accuracy is reduced. This is caused by the fact that its weighted average
operator may lead to the dispersion of the attention distribution, which
results in the relationship between adjacent signals ignored. To address this
issue, in this paper, we introduce relative-position-awareness self-attention
(RPSA). It not only maintains the global-range dependency modeling ability of
self-attention, but also improves the localness modeling ability. Because the
local window length of the original RPSA is fixed and sensitive to different
test data, here we propose Gaussian-based self-attention (GSA) whose window
length is learnable and adaptive to the test data automatically. We further
generalize GSA to a new residual Gaussian self-attention (resGSA) for the
performance improvement. We apply RPSA, GSA, and resGSA to Transformer-based
speech recognition respectively. Experimental results on the AISHELL-1 Mandarin
speech recognition corpus demonstrate the effectiveness of the proposed
methods. For example, the resGSA-Transformer achieves a character error rate
(CER) of 5.86% on the test set, which is relative 7.8% lower than that of the
SA-Transformer. Although the performance of the proposed resGSA-Transformer is
only slightly better than that of the RPSA-Transformer, it does not have to
tune the window length manually.
