  Skeleton-based human action recognition has recently attracted increasing
attention due to the popularity of 3D skeleton data. One main challenge lies in
the large view variations in captured human actions. We propose a novel view
adaptation scheme to automatically regulate observation viewpoints during the
occurrence of an action. Rather than re-positioning the skeletons based on a
human defined prior criterion, we design a view adaptive recurrent neural
network (RNN) with LSTM architecture, which enables the network itself to adapt
to the most suitable observation viewpoints from end to end. Extensive
experiment analyses show that the proposed view adaptive RNN model strives to
(1) transform the skeletons of various views to much more consistent viewpoints
and (2) maintain the continuity of the action rather than transforming every
frame to the same position with the same body orientation. Our model achieves
significant improvement over the state-of-the-art approaches on three benchmark
datasets.
