  Boundary-based instance segmentation has drawn much attention since of its
attractive efficiency. However, existing methods suffer from the difficulty in
long-distance regression. In this paper, we propose a coarse-to-fine module to
address the problem. Approximate boundary points are generated at the coarse
stage and then features of these points are sampled and fed to a refined
regressor for fine prediction. It is end-to-end trainable since differential
sampling operation is well supported in the module. Furthermore, we design a
holistic boundary-aware branch and introduce instance-agnostic supervision to
assist regression. Equipped with ResNet-101, our approach achieves 31.7\% mask
AP on COCO dataset with single-scale training and testing, outperforming the
baseline 1.3\% mask AP with less than 1\% additional parameters and GFLOPs.
Experiments also show that our proposed method achieves competitive performance
compared to existing boundary-based methods with a lightweight design and a
simple pipeline.
