  Since its introduction, UNet has been leading a variety of medical image
segmentation tasks. Although numerous follow-up studies have also been
dedicated to improving the performance of standard UNet, few have conducted
in-depth analyses of the underlying interest pattern of UNet in medical image
segmentation. In this paper, we explore the patterns learned in a UNet and
observe two important factors that potentially affect its performance: (i)
irrelative feature learned caused by asymmetric supervision; (ii) feature
redundancy in the feature map. To this end, we propose to balance the
supervision between encoder and decoder and reduce the redundant information in
the UNet. Specifically, we use the feature map that contains the most semantic
information (i.e., the last layer of the decoder) to provide additional
supervision to other blocks to provide additional supervision and reduce
feature redundancy by leveraging feature distillation. The proposed method can
be easily integrated into existing UNet architecture in a plug-and-play fashion
with negligible computational cost. The experimental results suggest that the
proposed method consistently improves the performance of standard UNets on four
medical image segmentation datasets. The code is available at
\url{https://github.com/ChongQingNoSubway/SelfReg-UNet}
