  Convolutional Neural Networks (CNNs) have shown to be powerful medical image
segmentation models. In this study, we address some of the main unresolved
issues regarding these models. Specifically, training of these models on small
medical image datasets is still challenging, with many studies promoting
techniques such as transfer learning. Moreover, these models are infamous for
producing over-confident predictions and for failing silently when presented
with out-of-distribution (OOD) data at test time. In this paper, we advocate
for multi-task learning, i.e., training a single model on several different
datasets, spanning several different organs of interest and different imaging
modalities. We show that not only a single CNN learns to automatically
recognize the context and accurately segment the organ of interest in each
context, but also that such a joint model often has more accurate and
better-calibrated predictions than dedicated models trained separately on each
dataset. Our experiments show that multi-task learning can outperform transfer
learning in medical image segmentation tasks. For detecting OOD data, we
propose a method based on spectral analysis of CNN feature maps. We show that
different datasets, representing different imaging modalities and/or different
organs of interest, have distinct spectral signatures, which can be used to
identify whether or not a test image is similar to the images used to train a
model. We show that this approach is far more accurate than OOD detection based
on prediction uncertainty. The methods proposed in this paper contribute
significantly to improving the accuracy and reliability of CNN-based medical
image segmentation models.
