  Semantic image segmentation is the process of labeling each pixel of an image
with its corresponding class. An encoder-decoder based approach, like U-Net and
its variants, is a popular strategy for solving medical image segmentation
tasks. To improve the performance of U-Net on various segmentation tasks, we
propose a novel architecture called DoubleU-Net, which is a combination of two
U-Net architectures stacked on top of each other. The first U-Net uses a
pre-trained VGG-19 as the encoder, which has already learned features from
ImageNet and can be transferred to another task easily. To capture more
semantic information efficiently, we added another U-Net at the bottom. We also
adopt Atrous Spatial Pyramid Pooling (ASPP) to capture contextual information
within the network. We have evaluated DoubleU-Net using four medical
segmentation datasets, covering various imaging modalities such as colonoscopy,
dermoscopy, and microscopy. Experiments on the MICCAI 2015 segmentation
challenge, the CVC-ClinicDB, the 2018 Data Science Bowl challenge, and the
Lesion boundary segmentation datasets demonstrate that the DoubleU-Net
outperforms U-Net and the baseline models. Moreover, DoubleU-Net produces more
accurate segmentation masks, especially in the case of the CVC-ClinicDB and
MICCAI 2015 segmentation challenge datasets, which have challenging images such
as smaller and flat polyps. These results show the improvement over the
existing U-Net model. The encouraging results, produced on various medical
image segmentation datasets, show that DoubleU-Net can be used as a strong
baseline for both medical image segmentation and cross-dataset evaluation
testing to measure the generalizability of Deep Learning (DL) models.
