  A widely used algorithm for transfer learning is fine-tuning, where a
pre-trained model is fine-tuned on a target task with a small amount of labeled
data. When the capacity of the pre-trained model is much larger than the size
of the target data set, fine-tuning is prone to overfitting and "memorizing"
the training labels. Hence, an important question is to regularize fine-tuning
and ensure its robustness to noise. To address this question, we begin by
analyzing the generalization properties of fine-tuning. We present a PAC-Bayes
generalization bound that depends on the distance traveled in each layer during
fine-tuning and the noise stability of the fine-tuned model. We empirically
measure these quantities. Based on the analysis, we propose regularized
self-labeling -- the interpolation between regularization and self-labeling
methods, including (i) layer-wise regularization to constrain the distance
traveled in each layer; (ii) self label-correction and label-reweighting to
correct mislabeled data points (that the model is confident) and reweight less
confident data points. We validate our approach on an extensive collection of
image and text data sets using multiple pre-trained model architectures. Our
approach improves baseline methods by 1.76% (on average) for seven image
classification tasks and 0.75% for a few-shot classification task. When the
target data set includes noisy labels, our approach outperforms baseline
methods by 3.56% on average in two noisy settings.
