  Supervised image segmentation assigns image voxels to a set of labels, as
defined by a specific labeling protocol. In this paper, we decompose
segmentation into two steps. The first step is what we call "primitive
segmentation", where voxels that form sub-parts (primitives) of the various
segmentation labels available in the training data, are grouped together. The
second step involves computing a protocol-specific label map based on the
primitive segmentation. Our core contribution is a novel loss function for the
first step, where a primitive segmentation model is trained. The proposed loss
function is the entropy of the (protocol-specific) "ground truth" label map
conditioned on the primitive segmentation. The conditional entropy loss enables
combining training datasets that have been manually labeled with different
protocols. Furthermore, as we show empirically, it facilitates an efficient
strategy for transfer learning via a lightweight protocol adaptation model that
can be trained with little manually labeled data. We apply the proposed
approach to the volumetric segmentation of brain MRI scans, where we achieve
promising results.
