  Image instance segmentation is a fundamental research topic in autonomous
driving, which is crucial for scene understanding and road safety. Advanced
learning-based approaches often rely on the costly 2D mask annotations for
training. In this paper, we present a more artful framework, LiDAR-guided
Weakly Supervised Instance Segmentation (LWSIS), which leverages the
off-the-shelf 3D data, i.e., Point Cloud, together with the 3D boxes, as
natural weak supervisions for training the 2D image instance segmentation
models. Our LWSIS not only exploits the complementary information in multimodal
data during training, but also significantly reduces the annotation cost of the
dense 2D masks. In detail, LWSIS consists of two crucial modules, Point Label
Assignment (PLA) and Graph-based Consistency Regularization (GCR). The former
module aims to automatically assign the 3D point cloud as 2D point-wise labels,
while the latter further refines the predictions by enforcing geometry and
appearance consistency of the multimodal data. Moreover, we conduct a secondary
instance segmentation annotation on the nuScenes, named nuInsSeg, to encourage
further research on multimodal perception tasks. Extensive experiments on the
nuInsSeg, as well as the large-scale Waymo, show that LWSIS can substantially
improve existing weakly supervised segmentation models by only involving 3D
data during training. Additionally, LWSIS can also be incorporated into 3D
object detectors like PointPainting to boost the 3D detection performance for
free. The code and dataset are available at https://github.com/Serenos/LWSIS.
