  Self-attention models such as Transformers, which can capture temporal
relationships without being limited by the distance between events, have given
competitive speech recognition results. However, we note the range of the
learned context increases from the lower to upper self-attention layers, whilst
acoustic events often happen within short time spans in a left-to-right order.
This leads to a question: for speech recognition, is a global view of the
entire sequence useful for the upper self-attention encoder layers in
Transformers? To investigate this, we train models with lower
self-attention/upper feed-forward layers encoders on Wall Street Journal and
Switchboard. Compared to baseline Transformers, no performance drop but minor
gains are observed. We further developed a novel metric of the diagonality of
attention matrices and found the learned diagonality indeed increases from the
lower to upper encoder self-attention layers. We conclude the global view is
unnecessary in training upper encoder layers.
