  In deep learning, transferring information from a pretrained network to a
downstream task by finetuning has many benefits. The choice of task head plays
an important role in fine-tuning, as the pretrained and downstream tasks are
usually different. Although there exist many different designs for finetuning,
a full understanding of when and why these algorithms work has been elusive. We
analyze how the choice of task head controls feature adaptation and hence
influences the downstream performance. By decomposing the learning dynamics of
adaptation, we find that the key aspect is the training accuracy and loss at
the beginning of finetuning, which determines the "energy" available for the
feature's adaptation. We identify a significant trend in the effect of changes
in this initial energy on the resulting features after fine-tuning.
Specifically, as the energy increases, the Euclidean and cosine distances
between the resulting and original features increase, while their dot products
(and the resulting features' norm) first increase and then decrease. Inspired
by this, we give several practical principles that lead to better downstream
performance. We analytically prove this trend in an overparamterized linear
setting and verify its applicability to different experimental settings.
