  As the application of deep learning has expanded to real-world problems with
insufficient volume of training data, transfer learning recently has gained
much attention as means of improving the performance in such small-data regime.
However, when existing methods are applied between heterogeneous architectures
and tasks, it becomes more important to manage their detailed configurations
and often requires exhaustive tuning on them for the desired performance. To
address the issue, we propose a novel transfer learning approach based on
meta-learning that can automatically learn what knowledge to transfer from the
source network to where in the target network. Given source and target
networks, we propose an efficient training scheme to learn meta-networks that
decide (a) which pairs of layers between the source and target networks should
be matched for knowledge transfer and (b) which features and how much knowledge
from each feature should be transferred. We validate our meta-transfer approach
against recent transfer learning methods on various datasets and network
architectures, on which our automated scheme significantly outperforms the
prior baselines that find "what and where to transfer" in a hand-crafted
manner.
