  Motion capture from a monocular video is fundamental and crucial for us
humans to naturally experience and interact with each other in Virtual Reality
(VR) and Augmented Reality (AR). However, existing methods still struggle with
challenging cases involving self-occlusion and complex poses due to the lack of
effective motion prior modeling. In this paper, we present a novel variational
motion prior (VMP) learning approach for video-based motion capture to resolve
the above issue. Instead of directly building the correspondence between the
video and motion domain, We propose to learn a generic latent space for
capturing the prior distribution of all natural motions, which serve as the
basis for subsequent video-based motion capture tasks. To improve the
generalization capacity of prior space, we propose a transformer-based
variational autoencoder pretrained over marker-based 3D mocap data, with a
novel style-mapping block to boost the generation quality. Afterward, a
separate video encoder is attached to the pretrained motion generator for
end-to-end fine-tuning over task-specific video datasets. Compared to existing
motion prior models, our VMP model serves as a motion rectifier that can
effectively reduce temporal jittering and failure modes in frame-wise pose
estimation, leading to temporally stable and visually realistic motion capture
results. Furthermore, our VMP-based framework models motion at sequence level
and can directly generate motion clips in the forward pass, achieving real-time
motion capture during inference. Extensive experiments over both public
datasets and in-the-wild videos have demonstrated the efficacy and
generalization capability of our framework.
