  In this paper we consider the task of image-guided depth completion where our
system must infer the depth at every pixel of an input image based on the image
content and a sparse set of depth measurements. We propose a novel approach
that builds upon the strengths of modern deep learning techniques and classical
optimization algorithms and significantly improves performance. The proposed
method replaces the final $1\times 1$ convolutional layer employed in most
depth completion networks with a least squares fitting module which computes
weights by fitting the implicit depth bases to the given sparse depth
measurements. In addition, we show how our proposed method can be naturally
extended to a multi-scale formulation for improved self-supervised training. We
demonstrate through extensive experiments on various datasets that our approach
achieves consistent improvements over state-of-the-art baseline methods with
small computational overhead.
