  In this paper, we tackle the problem of RGB-D semantic segmentation of indoor
images. We take advantage of deconvolutional networks which can predict
pixel-wise class labels, and develop a new structure for deconvolution of
multiple modalities. We propose a novel feature transformation network to
bridge the convolutional networks and deconvolutional networks. In the feature
transformation network, we correlate the two modalities by discovering common
features between them, as well as characterize each modality by discovering
modality specific features. With the common features, we not only closely
correlate the two modalities, but also allow them to borrow features from each
other to enhance the representation of shared information. With specific
features, we capture the visual patterns that are only visible in one modality.
The proposed network achieves competitive segmentation accuracy on NYU depth
dataset V1 and V2.
