  Missing data is a common problem in machine learning and in retrospective
imaging research it is often encountered in the form of missing imaging
modalities. We propose to take into account missing modalities in the design
and training of neural networks, to ensure that they are capable of providing
the best possible prediction even when multiple images are not available. The
proposed network combines three modifications to the standard 3D UNet
architecture: a training scheme with dropout of modalities, a multi-pathway
architecture with fusion layer in the final stage, and the separate
pre-training of these pathways. These modifications are evaluated incrementally
in terms of performance on full and missing data, using the BraTS multi-modal
segmentation challenge. The final model shows significant improvement with
respect to the state of the art on missing data and requires less memory during
training.
