  Learning-based multi-view stereo (MVS) has by far centered around 3D
convolution on cost volumes. Due to the high computation and memory consumption
of 3D CNN, the resolution of output depth is often considerably limited.
Different from most existing works dedicated to adaptive refinement of cost
volumes, we opt to directly optimize the depth value along each camera ray,
mimicking the range (depth) finding of a laser scanner. This reduces the MVS
problem to ray-based depth optimization which is much more light-weight than
full cost volume optimization. In particular, we propose RayMVSNet which learns
sequential prediction of a 1D implicit field along each camera ray with the
zero-crossing point indicating scene depth. This sequential modeling, conducted
based on transformer features, essentially learns the epipolar line search in
traditional multi-view stereo. We also devise a multi-task learning for better
optimization convergence and depth accuracy. Our method ranks top on both the
DTU and the Tanks \& Temples datasets over all previous learning-based methods,
achieving overall reconstruction score of 0.33mm on DTU and f-score of 59.48%
on Tanks & Temples.
