  End-to-end automatic speech recognition (ASR) models aim to learn a
generalised speech representation. However, there are limited tools available
to understand the internal functions and the effect of hierarchical
dependencies within the model architecture. It is crucial to understand the
correlations between the layer-wise representations, to derive insights on the
relationship between neural representations and performance.
  Previous investigations of network similarities using correlation analysis
techniques have not been explored for End-to-End ASR models. This paper
analyses and explores the internal dynamics between layers during training with
CNN, LSTM and Transformer based approaches using Canonical correlation analysis
(CCA) and centered kernel alignment (CKA) for the experiments. It was found
that neural representations within CNN layers exhibit hierarchical correlation
dependencies as layer depth increases but this is mostly limited to cases where
neural representation correlates more closely. This behaviour is not observed
in LSTM architecture, however there is a bottom-up pattern observed across the
training process, while Transformer encoder layers exhibit irregular
coefficiency correlation as neural depth increases. Altogether, these results
provide new insights into the role that neural architectures have upon speech
recognition performance. More specifically, these techniques can be used as
indicators to build better performing speech recognition models.
