  In this paper, we propose a pose grammar to tackle the problem of 3D human
pose estimation. Our model directly takes 2D pose as input and learns a
generalized 2D-3D mapping function. The proposed model consists of a base
network which efficiently captures pose-aligned features and a hierarchy of
Bi-directional RNNs (BRNN) on the top to explicitly incorporate a set of
knowledge regarding human body configuration (i.e., kinematics, symmetry, motor
coordination). The proposed model thus enforces high-level constraints over
human poses. In learning, we develop a pose sample simulator to augment
training samples in virtual camera views, which further improves our model
generalizability. We validate our method on public 3D human pose benchmarks and
propose a new evaluation protocol working on cross-view setting to verify the
generalization capability of different methods. We empirically observe that
most state-of-the-art methods encounter difficulty under such setting while our
method can well handle such challenges.
