  Transformer-based models have gained increasing popularity achieving
state-of-the-art performance in many research fields including speech
translation. However, Transformer's quadratic complexity with respect to the
input sequence length prevents its adoption as is with audio signals, which are
typically represented by long sequences. Current solutions resort to an initial
sub-optimal compression based on a fixed sampling of raw audio features.
Therefore, potentially useful linguistic information is not accessible to
higher-level layers in the architecture. To solve this issue, we propose
Speechformer, an architecture that, thanks to reduced memory usage in the
attention layers, avoids the initial lossy compression and aggregates
information only at a higher level according to more informed linguistic
criteria. Experiments on three language pairs (en->de/es/nl) show the efficacy
of our solution, with gains of up to 0.8 BLEU on the standard MuST-C corpus and
of up to 4.0 BLEU in a low resource scenario.
