  Generating coherent and natural movement is the key challenge in video
generation. This research proposes to condense video generation into a problem
of motion generation, to improve the expressiveness of motion and make video
generation more manageable. This can be achieved by breaking down the video
generation process into latent motion generation and video reconstruction. We
present a latent motion diffusion (LaMD) framework, which consists of a
motion-decomposed video autoencoder and a diffusion-based motion generator, to
implement this idea. Through careful design, the motion-decomposed video
autoencoder can compress patterns in movement into a concise latent motion
representation. Meanwhile, the diffusion-based motion generator is able to
efficiently generate realistic motion on a continuous latent space under
multi-modal conditions, at a cost that is similar to that of image diffusion
models. Results show that LaMD generates high-quality videos with a wide range
of motions, from stochastic dynamics to highly controllable movements. It
achieves new state-of-the-art performance on benchmark datasets, including
BAIR, Landscape and CATER-GENs, for Image-to-Video (I2V) and
Text-Image-to-Video (TI2V) generation. The source code of LaMD will be made
available soon.
