  Deep learning has been the answer to many machine learning problems during
the past two decades. However, it comes with two major constraints: dependency
on extensive labeled data and training costs. Transfer learning in deep
learning, known as Deep Transfer Learning (DTL), attempts to reduce such
dependency and costs by reusing an obtained knowledge from a source data/task
in training on a target data/task. Most applied DTL techniques are
network/model-based approaches. These methods reduce the dependency of deep
learning models on extensive training data and drastically decrease training
costs. As a result, researchers detected Covid-19 infection on chest X-Rays
with high accuracy at the beginning of the pandemic with minimal data using DTL
techniques. Also, the training cost reduction makes DTL viable on edge devices
with limited resources. Like any new advancement, DTL methods have their own
limitations, and a successful transfer depends on some adjustments for
different scenarios. In this paper, we review the definition and taxonomy of
deep transfer learning and well-known methods. Then we investigate the DTL
approaches by reviewing recent applied DTL techniques in the past five years.
Further, we review some experimental analyses of DTLs to learn the best
practice for applying DTL in different scenarios. Moreover, the limitations of
DTLs (catastrophic forgetting dilemma and overly biased pre-trained models) are
discussed, along with possible solutions and research trends.
