  The growing use of Machine Learning has produced significant advances in many
fields. For image-based tasks, however, the use of deep learning remains
challenging in small datasets. In this article, we review, evaluate and compare
the current state-of-the-art techniques in training neural networks to
elucidate which techniques work best for small datasets. We further propose a
path forward for the improvement of model accuracy in medical imaging
applications. We observed best results from one cycle training, discriminative
learning rates with gradual freezing and parameter modification after transfer
learning. We also established that when datasets are small, transfer learning
plays an important role beyond parameter initialization by reusing previously
learned features. Surprisingly we observed that there is little advantage in
using pre-trained networks in images from another part of the body compared to
Imagenet. On the contrary, if images from the same part of the body are
available then transfer learning can produce a significant improvement in
performance with as little as 50 images in the training data.
