  Motion recognition is a promising direction in computer vision, but the
training of video classification models is much harder than images due to
insufficient data and considerable parameters. To get around this, some works
strive to explore multimodal cues from RGB-D data. Although improving motion
recognition to some extent, these methods still face sub-optimal situations in
the following aspects: (i) Data augmentation, i.e., the scale of the RGB-D
datasets is still limited, and few efforts have been made to explore novel data
augmentation strategies for videos; (ii) Optimization mechanism, i.e., the
tightly space-time-entangled network structure brings more challenges to
spatiotemporal information modeling; And (iii) cross-modal knowledge fusion,
i.e., the high similarity between multimodal representations caused to
insufficient late fusion. To alleviate these drawbacks, we propose to improve
RGB-D-based motion recognition both from data and algorithm perspectives in
this paper. In more detail, firstly, we introduce a novel video data
augmentation method dubbed ShuffleMix, which acts as a supplement to MixUp, to
provide additional temporal regularization for motion recognition. Secondly, a
Unified Multimodal De-coupling and multi-stage Re-coupling framework, termed
UMDR, is proposed for video representation learning. Finally, a novel
cross-modal Complement Feature Catcher (CFCer) is explored to mine potential
commonalities features in multimodal information as the auxiliary fusion
stream, to improve the late fusion results. The seamless combination of these
novel designs forms a robust spatiotemporal representation and achieves better
performance than state-of-the-art methods on four public motion datasets.
Specifically, UMDR achieves unprecedented improvements of +4.5% on the Chalearn
IsoGD dataset. Our code is available at
https://github.com/zhoubenjia/MotionRGBD-PAMI.
