  Scene understanding has made tremendous progress over the past few years, as
data acquisition systems are now providing an increasing amount of data of
various modalities (point cloud, depth, RGB...). However, this improvement
comes at a large cost on computation resources and data annotation
requirements. To analyze geometric information and images jointly, many
approaches rely on both a 2D loss and 3D loss, requiring not only 2D per
pixel-labels but also 3D per-point labels. However, obtaining a 3D groundtruth
is challenging, time-consuming and error-prone. In this paper, we show that
image segmentation can benefit from 3D geometric information without requiring
a 3D groundtruth, by training the geometric feature extraction and the 2D
segmentation network jointly, in an end-to-end fashion, using only the 2D
segmentation loss. Our method starts by extracting a map of 3D features
directly from a provided point cloud by using a lightweight 3D neural network.
The 3D feature map, merged with the RGB image, is then used as an input to a
classical image segmentation network. Our method can be applied to many 2D
segmentation networks, improving significantly their performance with only a
marginal network weight increase and light input dataset requirements, since no
3D groundtruth is required.
