  One major goal of vision is to infer physical models of objects, surfaces,
and their layout from sensors. In this paper, we aim to interpret indoor scenes
from one RGBD image. Our representation encodes the layout of orthogonal walls
and the extent of objects, modeled with CAD-like 3D shapes. We parse both the
visible and occluded portions of the scene and all observable objects,
producing a complete 3D parse. Such a scene interpretation is useful for
robotics and visual reasoning, but difficult to produce due to the well-known
challenge of segmentation, the high degree of occlusion, and the diversity of
objects in indoor scenes. We take a data-driven approach, generating sets of
potential object regions, matching to regions in training images, and
transferring and aligning associated 3D models while encouraging fit to
observations and spatial consistency. We use support inference to aid
interpretation and propose a retrieval scheme that uses convolutional neural
networks (CNNs) to classify regions and retrieve objects with similar shapes.
We demonstrate the performance of our method on our newly annotated NYUd v2
dataset with detailed 3D shapes.
