  There is an increasing number of pre-trained deep neural network models.
However, it is still unclear how to effectively use these models for a new
task. Transfer learning, which aims to transfer knowledge from source tasks to
a target task, is an effective solution to this problem. Fine-tuning is a
popular transfer learning technique for deep neural networks where a few rounds
of training are applied to the parameters of a pre-trained model to adapt them
to a new task. Despite its popularity, in this paper, we show that fine-tuning
suffers from several drawbacks. We propose an adaptive fine-tuning approach,
called AdaFilter, which selects only a part of the convolutional filters in the
pre-trained model to optimize on a per-example basis. We use a recurrent gated
network to selectively fine-tune convolutional filters based on the activations
of the previous layer. We experiment with 7 public image classification
datasets and the results show that AdaFilter can reduce the average
classification error of the standard fine-tuning by 2.54%.
