  End-to-End automatic speech recognition (ASR) models aim to learn a
generalised speech representation to perform recognition. In this domain there
is little research to analyse internal representation dependencies and their
relationship to modelling approaches. This paper investigates cross-domain
language model dependencies within transformer architectures using SVCCA and
uses these insights to exploit modelling approaches. It was found that specific
neural representations within the transformer layers exhibit correlated
behaviour which impacts recognition performance.
  Altogether, this work provides analysis of the modelling approaches affecting
contextual dependencies and ASR performance, and can be used to create or adapt
better performing End-to-End ASR models and also for downstream tasks.
