  Multi-modal MR imaging is routinely used in clinical practice to diagnose and
investigate brain tumors by providing rich complementary information. Previous
multi-modal MRI segmentation methods usually perform modal fusion by
concatenating multi-modal MRIs at an early/middle stage of the network, which
hardly explores non-linear dependencies between modalities. In this work, we
propose a novel Nested Modality-Aware Transformer (NestedFormer) to explicitly
explore the intra-modality and inter-modality relationships of multi-modal MRIs
for brain tumor segmentation. Built on the transformer-based multi-encoder and
single-decoder structure, we perform nested multi-modal fusion for high-level
representations of different modalities and apply modality-sensitive gating
(MSG) at lower scales for more effective skip connections. Specifically, the
multi-modal fusion is conducted in our proposed Nested Modality-aware Feature
Aggregation (NMaFA) module, which enhances long-term dependencies within
individual modalities via a tri-orientated spatial-attention transformer, and
further complements key contextual information among modalities via a
cross-modality attention transformer. Extensive experiments on BraTS2020
benchmark and a private meningiomas segmentation (MeniSeg) dataset show that
the NestedFormer clearly outperforms the state-of-the-arts. The code is
available at https://github.com/920232796/NestedFormer.
