  Due to the compact and rich high-level representations offered,
skeleton-based human action recognition has recently become a highly active
research topic. Previous studies have demonstrated that investigating joint
relationships in spatial and temporal dimensions provides effective information
critical to action recognition. However, effectively encoding global
dependencies of joints during spatio-temporal feature extraction is still
challenging. In this paper, we introduce Action Capsule which identifies
action-related key joints by considering the latent correlation of joints in a
skeleton sequence. We show that, during inference, our end-to-end network pays
attention to a set of joints specific to each action, whose encoded
spatio-temporal features are aggregated to recognize the action. Additionally,
the use of multiple stages of action capsules enhances the ability of the
network to classify similar actions. Consequently, our network outperforms the
state-of-the-art approaches on the N-UCLA dataset and obtains competitive
results on the NTURGBD dataset. This is while our approach has significantly
lower computational requirements based on GFLOPs measurements.
