  We present SkeleTR, a new framework for skeleton-based action recognition. In
contrast to prior work, which focuses mainly on controlled environments, we
target more general scenarios that typically involve a variable number of
people and various forms of interaction between people. SkeleTR works with a
two-stage paradigm. It first models the intra-person skeleton dynamics for each
skeleton sequence with graph convolutions, and then uses stacked Transformer
encoders to capture person interactions that are important for action
recognition in general scenarios. To mitigate the negative impact of inaccurate
skeleton associations, SkeleTR takes relative short skeleton sequences as input
and increases the number of sequences. As a unified solution, SkeleTR can be
directly applied to multiple skeleton-based action tasks, including video-level
action classification, instance-level action detection, and group-level
activity recognition. It also enables transfer learning and joint training
across different action tasks and datasets, which result in performance
improvement. When evaluated on various skeleton-based action recognition
benchmarks, SkeleTR achieves the state-of-the-art performance.
