  We aim to tackle the interesting yet challenging problem of generating videos
of diverse and natural human motions from prescribed action categories. The key
issue lies in the ability to synthesize multiple distinct motion sequences that
are realistic in their visual appearances. It is achieved in this paper by a
two-step process that maintains internal 3D pose and shape representations,
action2motion and motion2video. Action2motion stochastically generates
plausible 3D pose sequences of a prescribed action category, which are
processed and rendered by motion2video to form 2D videos. Specifically, the Lie
algebraic theory is engaged in representing natural human motions following the
physical law of human kinematics; a temporal variational auto-encoder (VAE) is
developed that encourages diversity of output motions. Moreover, given an
additional input image of a clothed human character, an entire pipeline is
proposed to extract his/her 3D detailed shape, and to render in videos the
plausible motions from different views. This is realized by improving existing
methods to extract 3D human shapes and textures from single 2D images, rigging,
animating, and rendering to form 2D videos of human motions. It also
necessitates the curation and reannotation of 3D human motion datasets for
training purpose. Thorough empirical experiments including ablation study,
qualitative and quantitative evaluations manifest the applicability of our
approach, and demonstrate its competitiveness in addressing related tasks,
where components of our approach are compared favorably to the
state-of-the-arts.
