  Transferring knowledge from large source datasets is an effective way to
fine-tune the deep neural networks of the target task with a small sample size.
A great number of algorithms have been proposed to facilitate deep transfer
learning, and these techniques could be generally categorized into two groups -
Regularized Learning of the target task using models that have been pre-trained
from source datasets, and Multitask Learning with both source and target
datasets to train a shared backbone neural network. In this work, we aim to
improve the multitask paradigm for deep transfer learning via Cross-domain
Mixup (XMixup). While the existing multitask learning algorithms need to run
backpropagation over both the source and target datasets and usually consume a
higher gradient complexity, XMixup transfers the knowledge from source to
target tasks more efficiently: for every class of the target task, XMixup
selects the auxiliary samples from the source dataset and augments training
samples via the simple mixup strategy. We evaluate XMixup over six real world
transfer learning datasets. Experiment results show that XMixup improves the
accuracy by 1.9% on average. Compared with other state-of-the-art transfer
learning approaches, XMixup costs much less training time while still obtains
higher accuracy.
