  Human skeleton data has received increasing attention in action recognition
due to its background robustness and high efficiency. In skeleton-based action
recognition, graph convolutional network (GCN) has become the mainstream
method. This paper analyzes the fundamental factor for GCN-based models -- the
adjacency matrix. We notice that most GCN-based methods conduct their adjacency
matrix based on the human natural skeleton structure. Based on our former work
and analysis, we propose that the human natural skeleton structure adjacency
matrix is not proper for skeleton-based action recognition. We propose a new
adjacency matrix that abandons all rigid neighbor connections but lets the
model adaptively learn the relationships of joints. We conduct extensive
experiments and analysis with a validation model on two skeleton-based action
recognition datasets (NTURGBD60 and FineGYM). Comprehensive experimental
results and analysis reveals that 1) the most widely used human natural
skeleton structure adjacency matrix is unsuitable in skeleton-based action
recognition; 2) The proposed adjacency matrix is superior in model performance,
noise robustness and transferability.
