  Fine-tuning deep neural networks pre-trained on large scale datasets is one
of the most practical transfer learning paradigm given limited quantity of
training samples. To obtain better generalization, using the starting point as
the reference (SPAR), either through weights or features, has been successfully
applied to transfer learning as a regularizer. However, due to the domain
discrepancy between the source and target task, there exists obvious risk of
negative transfer in a straightforward manner of knowledge preserving. In this
paper, we propose a novel transfer learning algorithm, introducing the idea of
Target-awareness REpresentation Disentanglement (TRED), where the relevant
knowledge with respect to the target task is disentangled from the original
source model and used as a regularizer during fine-tuning the target model.
Specifically, we design two alternative methods, maximizing the Maximum Mean
Discrepancy (Max-MMD) and minimizing the mutual information (Min-MI), for the
representation disentanglement. Experiments on various real world datasets show
that our method stably improves the standard fine-tuning by more than 2% in
average. TRED also outperforms related state-of-the-art transfer learning
regularizers such as L2-SP, AT, DELTA, and BSS.
