  Transformer has achieved great success in the NLP field by composing various
advanced models like BERT and GPT. However, Transformer and its existing
variants may not be optimal in capturing token distances because the position
or distance embeddings used by these methods usually cannot keep the precise
information of real distances, which may not be beneficial for modeling the
orders and relations of contexts. In this paper, we propose DA-Transformer,
which is a distance-aware Transformer that can exploit the real distance. We
propose to incorporate the real distances between tokens to re-scale the raw
self-attention weights, which are computed by the relevance between attention
query and key. Concretely, in different self-attention heads the relative
distance between each pair of tokens is weighted by different learnable
parameters, which control the different preferences on long- or short-term
information of these heads. Since the raw weighted real distances may not be
optimal for adjusting self-attention weights, we propose a learnable sigmoid
function to map them into re-scaled coefficients that have proper ranges. We
first clip the raw self-attention weights via the ReLU function to keep
non-negativity and introduce sparsity, and then multiply them with the
re-scaled coefficients to encode real distance information into self-attention.
Extensive experiments on five benchmark datasets show that DA-Transformer can
effectively improve the performance of many tasks and outperform the vanilla
Transformer and its several variants.
