  Convolutional neural networks (CNNs) are extensively beneficial for medical
image processing. Medical images are plentiful, but there is a lack of
annotated data. Transfer learning is used to solve the problem of lack of
labeled data and grants CNNs better training capability. Transfer learning can
be used in many different medical applications; however, the model under
transfer should have the same size as the original network. Knowledge
distillation is recently proposed to transfer the knowledge of a model to
another one and can be useful to cover the shortcomings of transfer learning.
But some parts of the knowledge may not be distilled by knowledge distillation.
In this paper, a novel knowledge distillation using transfer learning is
proposed to transfer the whole knowledge of a model to another one. The
proposed method can be beneficial and practical for medical image analysis in
which a small number of labeled data are available. The proposed process is
tested for diabetic retinopathy classification. Simulation results demonstrate
that using the proposed method, knowledge of an extensive network can be
transferred to a smaller model.
