  Accurate stereo depth estimation plays a critical role in various 3D tasks in
both indoor and outdoor environments. Recently, learning-based multi-view
stereo methods have demonstrated competitive performance with a limited number
of views. However, in challenging scenarios, especially when building
cross-view correspondences is hard, these methods still cannot produce
satisfying results. In this paper, we study how to leverage a normal estimation
model and the predicted normal maps to improve the depth quality. We couple the
learning of a multi-view normal estimation module and a multi-view depth
estimation module. In addition, we propose a novel consistency loss to train an
independent consistency module that refines the depths from depth/normal pairs.
We find that the joint learning can improve both the prediction of normal and
depth, and the accuracy & smoothness can be further improved by enforcing the
consistency. Experiments on MVS, SUN3D, RGBD, and Scenes11 demonstrate the
effectiveness of our method and state-of-the-art performance.
