  Owing to the success of transformer models, recent works study their
applicability in 3D medical segmentation tasks. Within the transformer models,
the self-attention mechanism is one of the main building blocks that strives to
capture long-range dependencies. However, the self-attention operation has
quadratic complexity which proves to be a computational bottleneck, especially
in volumetric medical imaging, where the inputs are 3D with numerous slices. In
this paper, we propose a 3D medical image segmentation approach, named UNETR++,
that offers both high-quality segmentation masks as well as efficiency in terms
of parameters, compute cost, and inference speed. The core of our design is the
introduction of a novel efficient paired attention (EPA) block that efficiently
learns spatial and channel-wise discriminative features using a pair of
inter-dependent branches based on spatial and channel attention. Our spatial
attention formulation is efficient having linear complexity with respect to the
input sequence length. To enable communication between spatial and
channel-focused branches, we share the weights of query and key mapping
functions that provide a complimentary benefit (paired attention), while also
reducing the overall network parameters. Our extensive evaluations on five
benchmarks, Synapse, BTCV, ACDC, BRaTs, and Decathlon-Lung, reveal the
effectiveness of our contributions in terms of both efficiency and accuracy. On
Synapse, our UNETR++ sets a new state-of-the-art with a Dice Score of 87.2%,
while being significantly efficient with a reduction of over 71% in terms of
both parameters and FLOPs, compared to the best method in the literature. Code:
https://github.com/Amshaker/unetr_plus_plus.
