  Analyzing the similarity of internal representations within and across
different models has been an important technique for understanding the behavior
of deep neural networks. Most existing methods for analyzing the similarity
between representations of high dimensions, such as those based on Canonical
Correlation Analysis (CCA) and widely used Centered Kernel Alignment (CKA),
rely on statistical properties of the representations for a set of data points.
In this paper, we focus on transformer models and study the similarity of
representations between the hidden layers of individual transformers. In this
context, we show that a simple sample-wise cosine similarity metric is capable
of capturing the similarity and aligns with the complicated CKA. Our
experimental results on common transformers reveal that representations across
layers are positively correlated, albeit the similarity decreases when layers
are far apart. We then propose an aligned training approach to enhance the
similarity between internal representations, with trained models that enjoy the
following properties: (1) the last-layer classifier can be directly applied
right after any hidden layers, yielding intermediate layer accuracies much
higher than those under standard training, (2) the layer-wise accuracies
monotonically increase and reveal the minimal depth needed for the given task,
(3) when served as multi-exit models, they achieve on-par performance with
standard multi-exit architectures which consist of additional classifiers
designed for early exiting in shallow layers. To our knowledge, our work is the
first to show that one common classifier is sufficient for multi-exit models.
We conduct experiments on both vision and NLP tasks to demonstrate the
performance of the proposed aligned training.
