  Skeleton-based action recognition has recently received considerable
attention. Current approaches to skeleton-based action recognition are
typically formulated as one-hot classification tasks and do not fully exploit
the semantic relations between actions. For example, "make victory sign" and
"thumb up" are two actions of hand gestures, whose major difference lies in the
movement of hands. This information is agnostic from the categorical one-hot
encoding of action classes but could be unveiled from the action description.
Therefore, utilizing action description in training could potentially benefit
representation learning. In this work, we propose a Generative
Action-description Prompts (GAP) approach for skeleton-based action
recognition. More specifically, we employ a pre-trained large-scale language
model as the knowledge engine to automatically generate text descriptions for
body parts movements of actions, and propose a multi-modal training scheme by
utilizing the text encoder to generate feature vectors for different body parts
and supervise the skeleton encoder for action representation learning.
Experiments show that our proposed GAP method achieves noticeable improvements
over various baseline models without extra computation cost at inference. GAP
achieves new state-of-the-arts on popular skeleton-based action recognition
benchmarks, including NTU RGB+D, NTU RGB+D 120 and NW-UCLA. The source code is
available at https://github.com/MartinXM/GAP.
