  Over the last few years machine learning has demonstrated groundbreaking
results in many areas of medical image analysis, including segmentation. A key
assumption, however, is that the train- and test distributions match. We study
a realistic scenario where this assumption is clearly violated, namely
segmentation with missing input modalities. We describe two neural network
approaches that can handle a variable number of input modalities. The first is
modality dropout: a simple but surprisingly effective modification of the
training. The second is the unified representation network: a network
architecture that maps a variable number of input modalities into a unified
representation that can be used for downstream tasks such as segmentation. We
demonstrate that modality dropout makes a standard segmentation network
reasonably robust to missing modalities, but that the same network works even
better if trained on the unified representation.
