  Transformers have achieved state-of-the-art results across multiple NLP
tasks. However, the self-attention mechanism complexity scales quadratically
with the sequence length, creating an obstacle for tasks involving long
sequences, like in the speech domain. In this paper, we discuss the usefulness
of self-attention for Direct Speech Translation. First, we analyze the
layer-wise token contributions in the self-attention of the encoder, unveiling
local diagonal patterns. To prove that some attention weights are avoidable, we
propose to substitute the standard self-attention with a local efficient one,
setting the amount of context used based on the results of the analysis. With
this approach, our model matches the baseline performance, and improves the
efficiency by skipping the computation of those weights that standard attention
discards.
