  Parameter fine tuning is a transfer learning approach whereby learned
parameters from pre-trained source network are transferred to the target
network followed by fine-tuning. Prior research has shown that this approach is
capable of improving task performance. However, the impact of the ImageNet
pre-trained classification layer in parameter fine-tuning is mostly unexplored
in the literature. In this paper, we propose a fine-tuning approach with the
pre-trained classification layer. We employ layer-wise fine-tuning to determine
which layers should be frozen for optimal performance. Our empirical analysis
demonstrates that the proposed fine-tuning performs better than traditional
fine-tuning. This finding indicates that the pre-trained classification layer
holds less category-specific or more global information than believed earlier.
Thus, we hypothesize that the presence of this layer is crucial for growing
network depth to adapt better to a new task. Our study manifests that careful
normalization and scaling are essential for creating harmony between the
pre-trained and new layers for target domain adaptation. We evaluate the
proposed depth augmented networks for fine-tuning on several challenging
benchmark datasets and show that they can achieve higher classification
accuracy than contemporary transfer learning approaches.
