  We focus on the problem of semantic segmentation based on RGB-D data, with
emphasis on analyzing cluttered indoor scenes containing many instances from
many visual categories. Our approach is based on a parametric figure-ground
intensity and depth-constrained proposal process that generates spatial layout
hypotheses at multiple locations and scales in the image followed by a
sequential inference algorithm that integrates the proposals into a complete
scene estimate. Our contributions can be summarized as proposing the following:
(1) a generalization of parametric max flow figure-ground proposal methodology
to take advantage of intensity and depth information, in order to
systematically and efficiently generate the breakpoints of an underlying
spatial model in polynomial time, (2) new region description methods based on
second-order pooling over multiple features constructed using both intensity
and depth channels, (3) an inference procedure that can resolve conflicts in
overlapping spatial partitions, and handles scenes with a large number of
objects category instances, of very different scales, (4) extensive evaluation
of the impact of depth, as well as the effectiveness of a large number of
descriptors, both pre-designed and automatically obtained using deep learning,
in a difficult RGB-D semantic segmentation problem with 92 classes. We report
state of the art results in the challenging NYU Depth v2 dataset, extended for
RMRC 2013 Indoor Segmentation Challenge, where currently the proposed model
ranks first, with an average score of 24.61% and a number of 39 classes won.
Moreover, we show that by combining second-order and deep learning features,
over 15% relative accuracy improvements can be additionally achieved. In a
scene classification benchmark, our methodology further improves the state of
the art by 24%.
