  Accurate image segmentation plays a crucial role in medical image analysis,
yet it faces great challenges of various shapes, diverse sizes, and blurry
boundaries. To address these difficulties, square kernel-based encoder-decoder
architecture has been proposed and widely used, but its performance remains
still unsatisfactory. To further cope with these challenges, we present a novel
double-branch encoder architecture. Our architecture is inspired by two
observations: 1) Since the discrimination of features learned via square
convolutional kernels needs to be further improved, we propose to utilize
non-square vertical and horizontal convolutional kernels in the double-branch
encoder, so features learned by the two branches can be expected to complement
each other. 2) Considering that spatial attention can help models to better
focus on the target region in a large-sized image, we develop an attention loss
to further emphasize the segmentation on small-sized targets. Together, the
above two schemes give rise to a novel double-branch encoder segmentation
framework for medical image segmentation, namely Crosslink-Net. The experiments
validate the effectiveness of our model on four datasets. The code is released
at https://github.com/Qianyu1226/Crosslink-Net.
