  We present a novel framework named PlaneMVS for 3D plane reconstruction from
multiple input views with known camera poses. Most previous learning-based
plane reconstruction methods reconstruct 3D planes from single images, which
highly rely on single-view regression and suffer from depth scale ambiguity. In
contrast, we reconstruct 3D planes with a multi-view-stereo (MVS) pipeline that
takes advantage of multi-view geometry. We decouple plane reconstruction into a
semantic plane detection branch and a plane MVS branch. The semantic plane
detection branch is based on a single-view plane detection framework but with
differences. The plane MVS branch adopts a set of slanted plane hypotheses to
replace conventional depth hypotheses to perform plane sweeping strategy and
finally learns pixel-level plane parameters and its planar depth map. We
present how the two branches are learned in a balanced way, and propose a
soft-pooling loss to associate the outputs of the two branches and make them
benefit from each other. Extensive experiments on various indoor datasets show
that PlaneMVS significantly outperforms state-of-the-art (SOTA) single-view
plane reconstruction methods on both plane detection and 3D geometry metrics.
Our method even outperforms a set of SOTA learning-based MVS methods thanks to
the learned plane priors. To the best of our knowledge, this is the first work
on 3D plane reconstruction within an end-to-end MVS framework. Source code:
https://github.com/oppo-us-research/PlaneMVS.
