  Despite the success of fine-tuning pretrained language encoders like BERT for
downstream natural language understanding (NLU) tasks, it is still poorly
understood how neural networks change after fine-tuning. In this work, we use
centered kernel alignment (CKA), a method for comparing learned
representations, to measure the similarity of representations in task-tuned
models across layers. In experiments across twelve NLU tasks, we discover a
consistent block diagonal structure in the similarity of representations within
fine-tuned RoBERTa and ALBERT models, with strong similarity within clusters of
earlier and later layers, but not between them. The similarity of later layer
representations implies that later layers only marginally contribute to task
performance, and we verify in experiments that the top few layers of fine-tuned
Transformers can be discarded without hurting performance, even with no further
tuning.
