  In accelerated MRI reconstruction, the anatomy of a patient is recovered from
a set of under-sampled and noisy measurements. Deep learning approaches have
been proven to be successful in solving this ill-posed inverse problem and are
capable of producing very high quality reconstructions. However, current
architectures heavily rely on convolutions, that are content-independent and
have difficulties modeling long-range dependencies in images. Recently,
Transformers, the workhorse of contemporary natural language processing, have
emerged as powerful building blocks for a multitude of vision tasks. These
models split input images into non-overlapping patches, embed the patches into
lower-dimensional tokens and utilize a self-attention mechanism that does not
suffer from the aforementioned weaknesses of convolutional architectures.
However, Transformers incur extremely high compute and memory cost when 1) the
input image resolution is high and 2) when the image needs to be split into a
large number of patches to preserve fine detail information, both of which are
typical in low-level vision problems such as MRI reconstruction, having a
compounding effect. To tackle these challenges, we propose HUMUS-Net, a hybrid
architecture that combines the beneficial implicit bias and efficiency of
convolutions with the power of Transformer blocks in an unrolled and
multi-scale network. HUMUS-Net extracts high-resolution features via
convolutional blocks and refines low-resolution features via a novel
Transformer-based multi-scale feature extractor. Features from both levels are
then synthesized into a high-resolution output reconstruction. Our network
establishes new state of the art on the largest publicly available MRI dataset,
the fastMRI dataset. We further demonstrate the performance of HUMUS-Net on two
other popular MRI datasets and perform fine-grained ablation studies to
validate our design.
