  We study the generalization behavior of transfer learning of deep neural
networks (DNNs). We adopt the overparameterization perspective -- featuring
interpolation of the training data (i.e., approximately zero train error) and
the double descent phenomenon -- to explain the delicate effect of the transfer
learning setting on generalization performance. We study how the generalization
behavior of transfer learning is affected by the dataset size in the source and
target tasks, the number of transferred layers that are kept frozen in the
target DNN training, and the similarity between the source and target tasks. We
show that the test error evolution during the target DNN training has a more
significant double descent effect when the target training dataset is
sufficiently large. In addition, a larger source training dataset can yield a
slower target DNN training. Moreover, we demonstrate that the number of frozen
layers can determine whether the transfer learning is effectively
underparameterized or overparameterized and, in turn, this may induce a
freezing-wise double descent phenomenon that determines the relative success or
failure of learning. Also, we show that the double descent phenomenon may make
a transfer from a less related source task better than a transfer from a more
related source task. We establish our results using image classification
experiments with the ResNet, DenseNet and the vision transformer (ViT)
architectures.
