  Training a deep convolutional neural network (CNN) from scratch is difficult
because it requires a large amount of labeled training data and a great deal of
expertise to ensure proper convergence. A promising alternative is to fine-tune
a CNN that has been pre-trained using, for instance, a large set of labeled
natural images. However, the substantial differences between natural and
medical images may advise against such knowledge transfer. In this paper, we
seek to answer the following central question in the context of medical image
analysis: \emph{Can the use of pre-trained deep CNNs with sufficient
fine-tuning eliminate the need for training a deep CNN from scratch?} To
address this question, we considered 4 distinct medical imaging applications in
3 specialties (radiology, cardiology, and gastroenterology) involving
classification, detection, and segmentation from 3 different imaging
modalities, and investigated how the performance of deep CNNs trained from
scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner.
Our experiments consistently demonstrated that (1) the use of a pre-trained CNN
with adequate fine-tuning outperformed or, in the worst case, performed as well
as a CNN trained from scratch; (2) fine-tuned CNNs were more robust to the size
of training sets than CNNs trained from scratch; (3) neither shallow tuning nor
deep tuning was the optimal choice for a particular application; and (4) our
layer-wise fine-tuning scheme could offer a practical way to reach the best
performance for the application at hand based on the amount of available data.
