  We present an efficient text-to-video generation framework based on latent
diffusion models, termed MagicVideo. MagicVideo can generate smooth video clips
that are concordant with the given text descriptions. Due to a novel and
efficient 3D U-Net design and modeling video distributions in a low-dimensional
space, MagicVideo can synthesize video clips with 256x256 spatial resolution on
a single GPU card, which takes around 64x fewer computations than the Video
Diffusion Models (VDM) in terms of FLOPs. In specific, unlike existing works
that directly train video models in the RGB space, we use a pre-trained VAE to
map video clips into a low-dimensional latent space and learn the distribution
of videos' latent codes via a diffusion model. Besides, we introduce two new
designs to adapt the U-Net denoiser trained on image tasks to video data: a
frame-wise lightweight adaptor for the image-to-video distribution adjustment
and a directed temporal attention module to capture temporal dependencies
across frames. Thus, we can exploit the informative weights of convolution
operators from a text-to-image model for accelerating video training. To
ameliorate the pixel dithering in the generated videos, we also propose a novel
VideoVAE auto-encoder for better RGB reconstruction. We conduct extensive
experiments and demonstrate that MagicVideo can generate high-quality video
clips with either realistic or imaginary content. Refer to
\url{https://magicvideo.github.io/#} for more examples.
