  Accurate medical image segmentation is essential for diagnosis and treatment
planning of diseases. Convolutional Neural Networks (CNNs) have achieved
state-of-the-art performance for automatic medical image segmentation. However,
they are still challenged by complicated conditions where the segmentation
target has large variations of position, shape and scale, and existing CNNs
have a poor explainability that limits their application to clinical decisions.
In this work, we make extensive use of multiple attentions in a CNN
architecture and propose a comprehensive attention-based CNN (CA-Net) for more
accurate and explainable medical image segmentation that is aware of the most
important spatial positions, channels and scales at the same time. In
particular, we first propose a joint spatial attention module to make the
network focus more on the foreground region. Then, a novel channel attention
module is proposed to adaptively recalibrate channel-wise feature responses and
highlight the most relevant feature channels. Also, we propose a scale
attention module implicitly emphasizing the most salient feature maps among
multiple scales so that the CNN is adaptive to the size of an object. Extensive
experiments on skin lesion segmentation from ISIC 2018 and multi-class
segmentation of fetal MRI found that our proposed CA-Net significantly improved
the average segmentation Dice score from 87.77% to 92.08% for skin lesion,
84.79% to 87.08% for the placenta and 93.20% to 95.88% for the fetal brain
respectively compared with U-Net. It reduced the model size to around 15 times
smaller with close or even better accuracy compared with state-of-the-art
DeepLabv3+. In addition, it has a much higher explainability than existing
networks by visualizing the attention weight maps. Our code is available at
https://github.com/HiLab-git/CA-Net
