  Recent work has shown that the structure of convolutional neural networks
(CNNs) induces a strong prior that favors natural images. This prior, known as
a deep image prior (DIP), is an effective regularizer in inverse problems such
as image denoising and inpainting. We extend the concept of the DIP to depth
images. Given color images and noisy and incomplete target depth maps, we
optimize a randomly-initialized CNN model to reconstruct a depth map restored
by virtue of using the CNN network structure as a prior combined with a
view-constrained photo-consistency loss. This loss is computed using images
from a geometrically calibrated camera from nearby viewpoints. We apply this
deep depth prior for inpainting and refining incomplete and noisy depth maps
within both binocular and multi-view stereo pipelines. Our quantitative and
qualitative evaluation shows that our refined depth maps are more accurate and
complete, and after fusion, produces dense 3D models of higher quality.
