  We propose a novel system for unsupervised skeleton-based action recognition.
Given inputs of body keypoints sequences obtained during various movements, our
system associates the sequences with actions. Our system is based on an
encoder-decoder recurrent neural network, where the encoder learns a separable
feature representation within its hidden states formed by training the model to
perform prediction task. We show that according to such unsupervised training
the decoder and the encoder self-organize their hidden states into a feature
space which clusters similar movements into the same cluster and distinct
movements into distant clusters. Current state-of-the-art methods for action
recognition are strongly supervised, i.e., rely on providing labels for
training. Unsupervised methods have been proposed, however, they require camera
and depth inputs (RGB+D) at each time step. In contrast, our system is fully
unsupervised, does not require labels of actions at any stage, and can operate
with body keypoints input only. Furthermore, the method can perform on various
dimensions of body keypoints (2D or 3D) and include additional cues describing
movements. We evaluate our system on three extensive action recognition
benchmarks with different number of actions and examples. Our results
outperform prior unsupervised skeleton-based methods, unsupervised RGB+D based
methods on cross-view tests and while being unsupervised have similar
performance to supervised skeleton-based action recognition.
