  This paper presents Ske2Grid, a new representation learning framework for
improved skeleton-based action recognition. In Ske2Grid, we define a regular
convolution operation upon a novel grid representation of human skeleton, which
is a compact image-like grid patch constructed and learned through three novel
designs. Specifically, we propose a graph-node index transform (GIT) to
construct a regular grid patch through assigning the nodes in the skeleton
graph one by one to the desired grid cells. To ensure that GIT is a bijection
and enrich the expressiveness of the grid representation, an up-sampling
transform (UPT) is learned to interpolate the skeleton graph nodes for filling
the grid patch to the full. To resolve the problem when the one-step UPT is
aggressive and further exploit the representation capability of the grid patch
with increasing spatial size, a progressive learning strategy (PLS) is proposed
which decouples the UPT into multiple steps and aligns them to multiple paired
GITs through a compact cascaded design learned progressively. We construct
networks upon prevailing graph convolution networks and conduct experiments on
six mainstream skeleton-based action recognition datasets. Experiments show
that our Ske2Grid significantly outperforms existing GCN-based solutions under
different benchmark settings, without bells and whistles. Code and models are
available at https://github.com/OSVAI/Ske2Grid
