  Attention layers are an integral part of modern end-to-end automatic speech
recognition systems, for instance as part of the Transformer or Conformer
architecture. Attention is typically multi-headed, where each head has an
independent set of learned parameters and operates on the same input feature
sequence. The output of multi-headed attention is a fusion of the outputs from
the individual heads. We empirically analyze the diversity between
representations produced by the different attention heads and demonstrate that
the heads become highly correlated during the course of training. We
investigate a few approaches to increasing attention head diversity, including
using different attention mechanisms for each head and auxiliary training loss
functions to promote head diversity. We show that introducing
diversity-promoting auxiliary loss functions during training is a more
effective approach, and obtain WER improvements of up to 6% relative on the
Librispeech corpus. Finally, we draw a connection between the diversity of
attention heads and the similarity of the gradients of head parameters.
