  This paper addresses the problem of learning to complete a scene's depth from
sparse depth points and images of indoor scenes. Specifically, we study the
case in which the sparse depth is computed from a visual-inertial simultaneous
localization and mapping (VI-SLAM) system. The resulting point cloud has low
density, it is noisy, and has non-uniform spatial distribution, as compared to
the input from active depth sensors, e.g., LiDAR or Kinect. Since the VI-SLAM
produces point clouds only over textured areas, we compensate for the missing
depth of the low-texture surfaces by leveraging their planar structures and
their surface normals which is an important intermediate representation. The
pre-trained surface normal network, however, suffers from large performance
degradation when there is a significant difference in the viewing direction
(especially the roll angle) of the test image as compared to the trained ones.
To address this limitation, we use the available gravity estimate from the
VI-SLAM to warp the input image to the orientation prevailing in the training
dataset. This results in a significant performance gain for the surface normal
estimate, and thus the dense depth estimates. Finally, we show that our method
outperforms other state-of-the-art approaches both on training (ScanNet and
NYUv2) and testing (collected with Azure Kinect) datasets.
