  Convolutional Neural Networks (CNNs) are widely used for image classification
in a variety of fields, including medical imaging. While most studies deploy
cross-entropy as the loss function in such tasks, a growing number of
approaches have turned to a family of contrastive learning-based losses. Even
though performance metrics such as accuracy, sensitivity and specificity are
regularly used for the evaluation of CNN classifiers, the features that these
classifiers actually learn are rarely identified and their effect on the
classification performance on out-of-distribution test samples is
insufficiently explored. In this paper, motivated by the real-world task of
lung nodule classification, we investigate the features that a CNN learns when
trained and tested on different distributions of a synthetic dataset with
controlled modes of variation. We show that different loss functions lead to
different features being learned and consequently affect the generalization
ability of the classifier on unseen data. This study provides some important
insights into the design of deep learning solutions for medical imaging tasks.
