  Semantic scene understanding from point clouds is particularly challenging as
the points reflect only a sparse set of the underlying 3D geometry. Previous
works often convert point cloud into regular grids (e.g. voxels or bird-eye
view images), and resort to grid-based convolutions for scene understanding. In
this work, we introduce RfD-Net that jointly detects and reconstructs dense
object surfaces directly from raw point clouds. Instead of representing scenes
with regular grids, our method leverages the sparsity of point cloud data and
focuses on predicting shapes that are recognized with high objectness. With
this design, we decouple the instance reconstruction into global object
localization and local shape prediction. It not only eases the difficulty of
learning 2-D manifold surfaces from sparse 3D space, the point clouds in each
object proposal convey shape details that support implicit function learning to
reconstruct any high-resolution surfaces. Our experiments indicate that
instance detection and reconstruction present complementary effects, where the
shape prediction head shows consistent effects on improving object detection
with modern 3D proposal network backbones. The qualitative and quantitative
evaluations further demonstrate that our approach consistently outperforms the
state-of-the-arts and improves over 11 of mesh IoU in object reconstruction.
