  In inductive transfer learning, fine-tuning pre-trained convolutional
networks substantially outperforms training from scratch. When using
fine-tuning, the underlying assumption is that the pre-trained model extracts
generic features, which are at least partially relevant for solving the target
task, but would be difficult to extract from the limited amount of data
available on the target task. However, besides the initialization with the
pre-trained model and the early stopping, there is no mechanism in fine-tuning
for retaining the features learned on the source task. In this paper, we
investigate several regularization schemes that explicitly promote the
similarity of the final solution with the initial model. We show the benefit of
having an explicit inductive bias towards the initial model, and we eventually
recommend a simple $L^2$ penalty with the pre-trained model being a reference
as the baseline of penalty for transfer learning tasks.
