  Deep learning has made significant impacts on multi-view stereo systems.
State-of-the-art approaches typically involve building a cost volume, followed
by multiple 3D convolution operations to recover the input image's pixel-wise
depth. While such end-to-end learning of plane-sweeping stereo advances public
benchmarks' accuracy, they are typically very slow to compute. We present
\ouralg, a highly efficient multi-view stereo algorithm that seamlessly
integrates multi-view constraints into single-view networks via an attention
mechanism. Since \ouralg only builds on 2D convolutions, it is at least
$2\times$ faster than all the notable counterparts. Moreover, our algorithm
produces precise depth estimations and 3D reconstructions, achieving
state-of-the-art results on challenging benchmarks ScanNet, SUN3D, RGBD, and
the classical DTU dataset. our algorithm also out-performs all other algorithms
in the setting of inexact camera poses. Our code is released at
\url{https://github.com/zhenpeiyang/MVS2D}
