  Semantic reconstruction of indoor scenes refers to both scene understanding
and object reconstruction. Existing works either address one part of this
problem or focus on independent objects. In this paper, we bridge the gap
between understanding and reconstruction, and propose an end-to-end solution to
jointly reconstruct room layout, object bounding boxes and meshes from a single
image. Instead of separately resolving scene understanding and object
reconstruction, our method builds upon a holistic scene context and proposes a
coarse-to-fine hierarchy with three components: 1. room layout with camera
pose; 2. 3D object bounding boxes; 3. object meshes. We argue that
understanding the context of each component can assist the task of parsing the
others, which enables joint understanding and reconstruction. The experiments
on the SUN RGB-D and Pix3D datasets demonstrate that our method consistently
outperforms existing methods in indoor layout estimation, 3D object detection
and mesh reconstruction.
