  We propose a cost volume-based neural network for depth inference from
multi-view images. We demonstrate that building a cost volume pyramid in a
coarse-to-fine manner instead of constructing a cost volume at a fixed
resolution leads to a compact, lightweight network and allows us inferring high
resolution depth maps to achieve better reconstruction results. To this end, we
first build a cost volume based on uniform sampling of fronto-parallel planes
across the entire depth range at the coarsest resolution of an image. Then,
given current depth estimate, we construct new cost volumes iteratively on the
pixelwise depth residual to perform depth map refinement. While sharing similar
insight with Point-MVSNet as predicting and refining depth iteratively, we show
that working on cost volume pyramid can lead to a more compact, yet efficient
network structure compared with the Point-MVSNet on 3D points. We further
provide detailed analyses of the relation between (residual) depth sampling and
image resolution, which serves as a principle for building compact cost volume
pyramid. Experimental results on benchmark datasets show that our model can
perform 6x faster and has similar performance as state-of-the-art methods. Code
is available at https://github.com/JiayuYANG/CVP-MVSNet
