  We present a novel framework that concurrently tackles hand action
recognition and 3D future hand motion prediction. While previous works focus on
either recognition or prediction, we propose a generative Transformer VAE
architecture to jointly capture both aspects, facilitating realistic motion
prediction by leveraging the short-term hand motion and long-term action
consistency observed across timestamps. To ensure faithful representation of
the semantic dependency and different temporal granularity of hand pose and
action, our framework is decomposed into two cascaded VAE blocks. The lower
pose block models short-span poses, while the upper action block models
long-span action. These are connected by a mid-level feature that represents
sub-second series of hand poses. Our framework is trained across multiple
datasets, where pose and action blocks are trained separately to fully utilize
pose-action annotations of different qualities. Evaluations show that on
multiple datasets, the joint modeling of recognition and prediction improves
over separate solutions, and the semantic and temporal hierarchy enables
long-term pose and action modeling.
