  Given the prevalence of pre-trained contextualized representations in today's
NLP, there have been many efforts to understand what information they contain,
and why they seem to be universally successful. The most common approach to use
these representations involves fine-tuning them for an end task. Yet, how
fine-tuning changes the underlying embedding space is less studied. In this
work, we study the English BERT family and use two probing techniques to
analyze how fine-tuning changes the space. We hypothesize that fine-tuning
affects classification performance by increasing the distances between examples
associated with different labels. We confirm this hypothesis with carefully
designed experiments on five different NLP tasks. Via these experiments, we
also discover an exception to the prevailing wisdom that "fine-tuning always
improves performance". Finally, by comparing the representations before and
after fine-tuning, we discover that fine-tuning does not introduce arbitrary
changes to representations; instead, it adjusts the representations to
downstream tasks while largely preserving the original spatial structure of the
data points.
