  Deep learning is ubiquitous across many areas areas of computer vision. It
often requires large scale datasets for training before being fine-tuned on
small-to-medium scale problems. Activity, or, in other words, action
recognition, is one of many application areas of deep learning. While there
exist many Convolutional Neural Network architectures that work with the RGB
and optical flow frames, training on the time sequences of 3D body skeleton
joints is often performed via recurrent networks such as LSTM.
  In this paper, we propose a new representation which encodes sequences of 3D
body skeleton joints in texture-like representations derived from
mathematically rigorous kernel methods. Such a representation becomes the first
layer in a standard CNN network e.g., ResNet-50, which is then used in the
supervised domain adaptation pipeline to transfer information from the source
to target dataset. This lets us leverage the available Kinect-based data beyond
training on a single dataset and outperform simple fine-tuning on any two
datasets combined in a naive manner. More specifically, in this paper we
utilize the overlapping classes between datasets. We associate datapoints of
the same class via so-called commonality, known from the supervised domain
adaptation. We demonstrate state-of-the-art results on three publicly available
benchmarks.
