  Multi-modality magnetic resonance imaging (MRI) can provide complementary
information for computer-aided diagnosis. Traditional deep learning algorithms
are suitable for identifying specific anatomical structures segmenting lesions
and classifying diseases with magnetic resonance images. However, manual labels
are limited due to high expense, which hinders further improvement of model
accuracy. Self-supervised learning (SSL) can effectively learn feature
representations from unlabeled data by pre-training and is demonstrated to be
effective in natural image analysis. Most SSL methods ignore the similarity of
multi-modality MRI, leading to model collapse. This limits the efficiency of
pre-training, causing low accuracy in downstream segmentation and
classification tasks. To solve this challenge, we establish and validate a
multi-modality MRI masked autoencoder consisting of hybrid mask pattern (HMP)
and pyramid barlow twin (PBT) module for SSL on multi-modality MRI analysis.
The HMP concatenates three masking steps forcing the SSL to learn the semantic
connections of multi-modality images by reconstructing the masking patches. We
have proved that the proposed HMP can avoid model collapse. The PBT module
exploits the pyramidal hierarchy of the network to construct barlow twin loss
between masked and original views, aligning the semantic representations of
image patches at different vision scales in latent space. Experiments on
BraTS2023, PI-CAI, and lung gas MRI datasets further demonstrate the
superiority of our framework over the state-of-the-art. The performance of the
segmentation and classification is substantially enhanced, supporting the
accurate detection of small lesion areas. The code is available at
https://github.com/LinxuanHan/M2-MAE.
