  In a research context, image acquisition will often involve a pre-defined
static protocol and the data will be of high quality. If we are to build
applications that work in hospitals without significant operational changes in
care delivery, algorithms should be designed to cope with the available data in
the best possible way. In a clinical environment, imaging protocols are highly
flexible, with MRI sequences commonly missing appropriate sequence labeling
(e.g. T1, T2, FLAIR). To this end we introduce PIMMS, a Permutation Invariant
Multi-Modal Segmentation technique that is able to perform inference over sets
of MRI scans without using modality labels. We present results which show that
our convolutional neural network can, in some settings, outperform a baseline
model which utilizes modality labels, and achieve comparable performance
otherwise.
