  Recent progress on action recognition has mainly focused on RGB and optical
flow features. In this paper, we approach the problem of joint-based action
recognition. Unlike other modalities, constellation of joints and their motion
generate models with succinct human motion information for activity
recognition. We present a new model for joint-based action recognition, which
first extracts motion features from each joint separately through a shared
motion encoder before performing collective reasoning. Our joint selector
module re-weights the joint information to select the most discriminative
joints for the task. We also propose a novel joint-contrastive loss that pulls
together groups of joint features which convey the same action. We strengthen
the joint-based representations by using a geometry-aware data augmentation
technique which jitters pose heatmaps while retaining the dynamics of the
action. We show large improvements over the current state-of-the-art
joint-based approaches on JHMDB, HMDB, Charades, AVA action recognition
datasets. A late fusion with RGB and Flow-based approaches yields additional
improvements. Our model also outperforms the existing baseline on Mimetics, a
dataset with out-of-context actions.
