  Transformer-based models have been achieving state-of-the-art results in
several fields of Natural Language Processing. However, its direct application
to speech tasks is not trivial. The nature of this sequences carries problems
such as long sequence lengths and redundancy between adjacent tokens.
Therefore, we believe that regular self-attention mechanism might not be well
suited for it.
  Different approaches have been proposed to overcome these problems, such as
the use of efficient attention mechanisms. However, the use of these methods
usually comes with a cost, which is a performance reduction caused by
information loss. In this study, we present the Multiformer, a
Transformer-based model which allows the use of different attention mechanisms
on each head. By doing this, the model is able to bias the self-attention
towards the extraction of more diverse token interactions, and the information
loss is reduced. Finally, we perform an analysis of the head contributions, and
we observe that those architectures where all heads relevance is uniformly
distributed obtain better results. Our results show that mixing attention
patterns along the different heads and layers outperforms our baseline by up to
0.7 BLEU.
