  Previous methods for skeleton-based gesture recognition mostly arrange the
skeleton sequence into a pseudo picture or spatial-temporal graph and apply
deep Convolutional Neural Network (CNN) or Graph Convolutional Network (GCN)
for feature extraction. Although achieving superior results, these methods have
inherent limitations in dynamically capturing local features of interactive
hand parts, and the computing efficiency still remains a serious issue. In this
work, the self-attention mechanism is introduced to alleviate this problem.
Considering the hierarchical structure of hand joints, we propose an efficient
hierarchical self-attention network (HAN) for skeleton-based gesture
recognition, which is based on pure self-attention without any CNN, RNN or GCN
operators. Specifically, the joint self-attention module is used to capture
spatial features of fingers, the finger self-attention module is designed to
aggregate features of the whole hand. In terms of temporal features, the
temporal self-attention module is utilized to capture the temporal dynamics of
the fingers and the entire hand. Finally, these features are fused by the
fusion self-attention module for gesture classification. Experiments show that
our method achieves competitive results on three gesture recognition datasets
with much lower computational complexity.
