  Accelerated multi-modal magnetic resonance (MR) imaging is a new and
effective solution for fast MR imaging, providing superior performance in
restoring the target modality from its undersampled counterpart with guidance
from an auxiliary modality. However, existing works simply combine the
auxiliary modality as prior information, lacking in-depth investigations on the
potential mechanisms for fusing different modalities. Further, they usually
rely on the convolutional neural networks (CNNs), which is limited by the
intrinsic locality in capturing the long-distance dependency. To this end, we
propose a multi-modal transformer (MTrans), which is capable of transferring
multi-scale features from the target modality to the auxiliary modality, for
accelerated MR imaging. To capture deep multi-modal information, our MTrans
utilizes an improved multi-head attention mechanism, named cross attention
module, which absorbs features from the auxiliary modality that contribute to
the target modality. Our framework provides three appealing benefits: (i) Our
MTrans use an improved transformers for multi-modal MR imaging, affording more
global information compared with existing CNN-based methods. (ii) A new cross
attention module is proposed to exploit the useful information in each modality
at different scales. The small patch in the target modality aims to keep more
fine details, the large patch in the auxiliary modality aims to obtain
high-level context features from the larger region and supplement the target
modality effectively. (iii) We evaluate MTrans with various accelerated
multi-modal MR imaging tasks, e.g., MR image reconstruction and
super-resolution, where MTrans outperforms state-of-the-art methods on fastMRI
and real-world clinical datasets.
