  Gesture recognition is getting more and more popular due to various
application possibilities in human-machine interaction. Existing multi-modal
gesture recognition systems take multi-modal data as input to improve accuracy,
but such methods require more modality sensors, which will greatly limit their
application scenarios. Therefore we propose an end-to-end multi-task learning
framework in training 2D convolutional neural networks. The framework can use
the depth modality to improve accuracy during training and save costs by using
only RGB modality during inference. Our framework is trained to learn a
representation for multi-task learning: gesture segmentation and gesture
recognition. Depth modality contains the prior information for the location of
the gesture. Therefore it can be used as the supervision for gesture
segmentation. A plug-and-play module named Multi-Scale-Decoder is designed to
realize gesture segmentation, which contains two sub-decoder. It is used in the
lower stage and higher stage respectively, and can help the network pay
attention to key target areas, ignore irrelevant information, and extract more
discriminant features. Additionally, the MSD module and depth modality are only
used in the training stage to improve gesture recognition performance. Only RGB
modality and network without MSD are required during inference. Experimental
results on three public gesture recognition datasets show that our proposed
method provides superior performance compared with existing gesture recognition
frameworks. Moreover, using the proposed plug-and-play MSD in other 2D
CNN-based frameworks also get an excellent accuracy improvement.
