  Separating overlapped nuclei is a major challenge in histopathology image
analysis. Recently published approaches have achieved promising overall
performance on nuclei segmentation; however, their performance on separating
overlapped nuclei is quite limited. To address the issue, we propose a novel
multitask learning network with a bending loss regularizer to separate
overlapped nuclei accurately. The newly proposed multitask learning
architecture enhances the generalization by learning shared representation from
three tasks: instance segmentation, nuclei distance map prediction, and
overlapped nuclei distance map prediction. The proposed bending loss defines
high penalties to concave contour points with large curvatures, and applies
small penalties to convex contour points with small curvatures. Minimizing the
bending loss avoids generating contours that encompass multiple nuclei. In
addition, two new quantitative metrics, Aggregated Jaccard Index of overlapped
nuclei (AJIO) and Accuracy of overlapped nuclei (ACCO), are designed for the
evaluation of overlapped nuclei segmentation. We validate the proposed approach
on the CoNSeP and MoNuSegv1 datasets using seven quantitative metrics:
Aggregate Jaccard Index, Dice, Segmentation Quality, Recognition Quality,
Panoptic Quality, AJIO, and ACCO. Extensive experiments demonstrate that the
proposed Bend-Net outperforms eight state-of-the-art approaches.
