  Recent advances in learning reusable motion priors have demonstrated their
effectiveness in generating naturalistic behaviors. In this paper, we propose a
new learning framework in this paradigm for controlling physics-based
characters with significantly improved motion quality and diversity over
existing state-of-the-art methods. The proposed method uses reinforcement
learning (RL) to initially track and imitate life-like movements from
unstructured motion clips using the discrete information bottleneck, as adopted
in the Vector Quantized Variational AutoEncoder (VQ-VAE). This structure
compresses the most relevant information from the motion clips into a compact
yet informative latent space, i.e., a discrete space over vector quantized
codes. By sampling codes in the space from a trained categorical prior
distribution, high-quality life-like behaviors can be generated, similar to the
usage of VQ-VAE in computer vision. Although this prior distribution can be
trained with the supervision of the encoder's output, it follows the original
motion clip distribution in the dataset and could lead to imbalanced behaviors
in our setting. To address the issue, we further propose a technique named
prior shifting to adjust the prior distribution using curiosity-driven RL. The
outcome distribution is demonstrated to offer sufficient behavioral diversity
and significantly facilitates upper-level policy learning for downstream tasks.
We conduct comprehensive experiments using humanoid characters on two
challenging downstream tasks, sword-shield striking and two-player boxing game.
Our results demonstrate that the proposed framework is capable of controlling
the character to perform considerably high-quality movements in terms of
behavioral strategies, diversity, and realism. Videos, codes, and data are
available at https://tencent-roboticsx.github.io/NCP/.
