  We study the evolution of latent space in fine-tuned NLP models. Different
from the commonly used probing-framework, we opt for an unsupervised method to
analyze representations. More specifically, we discover latent concepts in the
representational space using hierarchical clustering. We then use an alignment
function to gauge the similarity between the latent space of a pre-trained
model and its fine-tuned version. We use traditional linguistic concepts to
facilitate our understanding and also study how the model space transforms
towards task-specific information. We perform a thorough analysis, comparing
pre-trained and fine-tuned models across three models and three downstream
tasks. The notable findings of our work are: i) the latent space of the higher
layers evolve towards task-specific concepts, ii) whereas the lower layers
retain generic concepts acquired in the pre-trained model, iii) we discovered
that some concepts in the higher layers acquire polarity towards the output
class, and iv) that these concepts can be used for generating adversarial
triggers.
