  We propose a novel method applicable in many scene understanding problems
that adapts the Monte Carlo Tree Search (MCTS) algorithm, originally designed
to learn to play games of high-state complexity. From a generated pool of
proposals, our method jointly selects and optimizes proposals that minimize the
objective term. In our first application for floor plan reconstruction from
point clouds, our method selects and refines the room proposals, modelled as 2D
polygons, by optimizing on an objective function combining the fitness as
predicted by a deep network and regularizing terms on the room shapes. We also
introduce a novel differentiable method for rendering the polygonal shapes of
these proposals. Our evaluations on the recent and challenging Structured3D and
Floor-SP datasets show significant improvements over the state-of-the-art,
without imposing hard constraints nor assumptions on the floor plan
configurations. In our second application, we extend our approach to
reconstruct general 3D room layouts from a color image and obtain accurate room
layouts. We also show that our differentiable renderer can easily be extended
for rendering 3D planar polygons and polygon embeddings. Our method shows high
performance on the Matterport3D-Layout dataset, without introducing hard
constraints on room layout configurations.
