  The success of deep learning algorithms generally depends on large-scale
data, while humans appear to have inherent ability of knowledge transfer, by
recognizing and applying relevant knowledge from previous learning experiences
when encountering and solving unseen tasks. Such an ability to acquire and
reuse knowledge is known as transferability in deep learning. It has formed the
long-term quest towards making deep learning as data-efficient as human
learning, and has been motivating fruitful design of more powerful deep
learning algorithms. We present this survey to connect different isolated areas
in deep learning with their relation to transferability, and to provide a
unified and complete view to investigating transferability through the whole
lifecycle of deep learning. The survey elaborates the fundamental goals and
challenges in parallel with the core principles and methods, covering recent
cornerstones in deep architectures, pre-training, task adaptation and domain
adaptation. This highlights unanswered questions on the appropriate objectives
for learning transferable knowledge and for adapting the knowledge to new tasks
and domains, avoiding catastrophic forgetting and negative transfer. Finally,
we implement a benchmark and an open-source library, enabling a fair evaluation
of deep learning methods in terms of transferability.
