  In this paper, we propose our Correlation For Completion Network (CFCNet), an
end-to-end deep learning model that uses the correlation between two data
sources to perform sparse depth completion. CFCNet learns to capture, to the
largest extent, the semantically correlated features between RGB and depth
information. Through pairs of image pixels and the visible measurements in a
sparse depth map, CFCNet facilitates feature-level mutual transformation of
different data sources. Such a transformation enables CFCNet to predict
features and reconstruct data of missing depth measurements according to their
corresponding, transformed RGB features. We extend canonical correlation
analysis to a 2D domain and formulate it as one of our training objectives
(i.e. 2d deep canonical correlation, or "2D2CCA loss"). Extensive experiments
validate the ability and flexibility of our CFCNet compared to the
state-of-the-art methods on both indoor and outdoor scenes with different
real-life sparse patterns. Codes are available at:
https://github.com/choyingw/CFCNet.
