  Transformer-based deep neural networks have achieved great success in various
sequence applications due to their powerful ability to model long-range
dependency. The key module of Transformer is self-attention (SA) which extracts
features from the entire sequence regardless of the distance between positions.
Although SA helps Transformer performs particularly well on long-range tasks,
SA requires quadratic computation and memory complexity with the input sequence
length. Recently, attention map reuse, which groups multiple SA layers to share
one attention map, has been proposed and achieved significant speedup for
speech recognition models. In this paper, we provide a comprehensive study on
attention map reuse focusing on its ability to accelerate inference. We compare
the method with other SA compression techniques and conduct a breakdown
analysis of its advantages for a long sequence. We demonstrate the
effectiveness of attention map reuse by measuring the latency on both CPU and
GPU platforms.
