  Convolutional Neural Networks (CNNs) have revolutionized performances in
several machine learning tasks such as image classification, object tracking,
and keyword spotting. However, given that they contain a large number of
parameters, their direct applicability into low resource tasks is not
straightforward. In this work, we experiment with an application of CNN models
to gastrointestinal landmark classification with only a few thousands of
training samples through transfer learning. As in a standard transfer learning
approach, we train CNNs on a large external corpus, followed by representation
extraction for the medical images. Finally, a classifier is trained on these
CNN representations. However, given that several variants of CNNs exist, the
choice of CNN is not obvious. To address this, we develop a novel metric that
can be used to predict test performances, given CNN representations on the
training set. Not only we demonstrate the superiority of the CNN based transfer
learning approach against an assembly of knowledge driven features, but the
proposed metric also carries an 87% correlation with the test set performances
as obtained using various CNN representations.
