  Character animation in real-world scenarios necessitates a variety of
constraints, such as trajectories, key-frames, interactions, etc. Existing
methodologies typically treat single or a finite set of these constraint(s) as
separate control tasks. They are often specialized, and the tasks they address
are rarely extendable or customizable. We categorize these as solutions to the
close-set motion control problem. In response to the complexity of practical
motion control, we propose and attempt to solve the open-set motion control
problem. This problem is characterized by an open and fully customizable set of
motion control tasks. To address this, we introduce a new paradigm,
programmable motion generation. In this paradigm, any given motion control task
is broken down into a combination of atomic constraints. These constraints are
then programmed into an error function that quantifies the degree to which a
motion sequence adheres to them. We utilize a pre-trained motion generation
model and optimize its latent code to minimize the error function of the
generated motion. Consequently, the generated motion not only inherits the
prior of the generative model but also satisfies the required constraints.
Experiments show that we can generate high-quality motions when addressing a
wide range of unseen tasks. These tasks encompass motion control by motion
dynamics, geometric constraints, physical laws, interactions with scenes,
objects or the character own body parts, etc. All of these are achieved in a
unified approach, without the need for ad-hoc paired training data collection
or specialized network designs. During the programming of novel tasks, we
observed the emergence of new skills beyond those of the prior model. With the
assistance of large language models, we also achieved automatic programming. We
hope that this work will pave the way for the motion control of general AI
agents.
