  Overall survival (OS) time is one of the most important evaluation indices
for gliomas situations. Multimodal Magnetic Resonance Imaging (MRI) scans play
an important role in the study of glioma prognosis OS time. Several deep
learning-based methods are proposed for the OS time prediction on multi-modal
MRI problems. However, these methods usually fuse multi-modal information at
the beginning or at the end of the deep learning networks and lack the fusion
of features from different scales. In addition, the fusion at the end of
networks always adapts global with global (eg. fully connected after
concatenation of global average pooling output) or local with local (eg.
bilinear pooling), which loses the information of local with global. In this
paper, we propose a novel method for multi-modal OS time prediction of brain
tumor patients, which contains an improved nonlocal features fusion module
introduced on different scales. Our method obtains a relative 8.76% improvement
over the current state-of-art method (0.6989 vs. 0.6426 on accuracy). Extensive
testing demonstrates that our method could adapt to situations with missing
modalities. The code is available at
https://github.com/TangWen920812/mmmna-net.
