  Indoor scene images usually contain scattered objects and various scene
layouts, which make RGB-D scene classification a challenging task. Existing
methods still have limitations for classifying scene images with great spatial
variability. Thus, how to extract local patch-level features effectively using
only image labels is still an open problem for RGB-D scene recognition. In this
paper, we propose an efficient framework for RGB-D scene recognition, which
adaptively selects important local features to capture the great spatial
variability of scene images. Specifically, we design a differentiable local
feature selection (DLFS) module, which can extract the appropriate number of
key local scenerelated features. Discriminative local theme-level and
object-level representations can be selected with the DLFS module from the
spatially-correlated multi-modal RGB-D features. We take advantage of the
correlation between RGB and depth modalities to provide more cues for selecting
local features. To ensure that discriminative local features are selected, the
variational mutual information maximization loss is proposed. Additionally, the
DLFS module can be easily extended to select local features of different
scales. By concatenating the local-orderless and global structured multi-modal
features, the proposed framework can achieve state-of-the-art performance on
public RGB-D scene recognition datasets.
