  We present a novel approach to reconstructing lightweight, CAD-based
representations of scanned 3D environments from commodity RGB-D sensors. Our
key idea is to jointly optimize for both CAD model alignments as well as layout
estimations of the scanned scene, explicitly modeling inter-relationships
between objects-to-objects and objects-to-layout. Since object arrangement and
scene layout are intrinsically coupled, we show that treating the problem
jointly significantly helps to produce globally-consistent representations of a
scene. Object CAD models are aligned to the scene by establishing dense
correspondences between geometry, and we introduce a hierarchical layout
prediction approach to estimate layout planes from corners and edges of the
scene.To this end, we propose a message-passing graph neural network to model
the inter-relationships between objects and layout, guiding generation of a
globally object alignment in a scene. By considering the global scene layout,
we achieve significantly improved CAD alignments compared to state-of-the-art
methods, improving from 41.83% to 58.41% alignment accuracy on SUNCG and from
50.05% to 61.24% on ScanNet, respectively. The resulting CAD-based
representations makes our method well-suited for applications in content
creation such as augmented- or virtual reality.
