  Pixel-wise clean annotation is necessary for fully-supervised semantic
segmentation, which is laborious and expensive to obtain. In this paper, we
propose a weakly supervised 2D semantic segmentation model by incorporating
sparse bounding box labels with available 3D information, which is much easier
to obtain with advanced sensors. We manually labeled a subset of the 2D-3D
Semantics(2D-3D-S) dataset with bounding boxes, and introduce our 2D-3D
inference module to generate accurate pixel-wise segment proposal masks. Guided
by 3D information, we first generate a point cloud of objects and calculate
objectness probability score for each point. Then we project the point cloud
with objectness probabilities back to 2D images followed by a refinement step
to obtain segment proposals, which are treated as pseudo labels to train a
semantic segmentation network. Our method works in a recursive manner to
gradually refine the above-mentioned segment proposals. Extensive experimental
results on the 2D-3D-S dataset show that the proposed method can generate
accurate segment proposals when bounding box labels are available on only a
small subset of training images. Performance comparison with recent
state-of-the-art methods further illustrates the effectiveness of our method.
