  In this paper, we propose a robust and efficient end-to-end non-local spatial
propagation network for depth completion. The proposed network takes RGB and
sparse depth images as inputs and estimates non-local neighbors and their
affinities of each pixel, as well as an initial depth map with pixel-wise
confidences. The initial depth prediction is then iteratively refined by its
confidence and non-local spatial propagation procedure based on the predicted
non-local neighbors and corresponding affinities. Unlike previous algorithms
that utilize fixed-local neighbors, the proposed algorithm effectively avoids
irrelevant local neighbors and concentrates on relevant non-local neighbors
during propagation. In addition, we introduce a learnable affinity
normalization to better learn the affinity combinations compared to
conventional methods. The proposed algorithm is inherently robust to the
mixed-depth problem on depth boundaries, which is one of the major issues for
existing depth estimation/completion algorithms. Experimental results on indoor
and outdoor datasets demonstrate that the proposed algorithm is superior to
conventional algorithms in terms of depth completion accuracy and robustness to
the mixed-depth problem. Our implementation is publicly available on the
project page.
