  This paper presents a learning-based method for multi-view depth estimation
from posed images. Our core idea is a "learning-to-optimize" paradigm that
iteratively indexes a plane-sweeping cost volume and regresses the depth map
via a convolutional Gated Recurrent Unit (GRU). Since the cost volume plays a
paramount role in encoding the multi-view geometry, we aim to improve its
construction both at pixel- and frame- levels. At the pixel level, we propose
to break the symmetry of the Siamese network (which is typically used in MVS to
extract image features) by introducing a transformer block to the reference
image (but not to the source images). Such an asymmetric volume allows the
network to extract global features from the reference image to predict its
depth map. Given potential inaccuracies in the poses between reference and
source images, we propose to incorporate a residual pose network to correct the
relative poses. This essentially rectifies the cost volume at the frame level.
We conduct extensive experiments on real-world MVS datasets and show that our
method achieves state-of-the-art performance in terms of both within-dataset
evaluation and cross-dataset generalization.
