  ControlNets are widely used for adding spatial control to text-to-image
diffusion models with different conditions, such as depth maps,
scribbles/sketches, and human poses. However, when it comes to controllable
video generation, ControlNets cannot be directly integrated into new backbones
due to feature space mismatches, and training ControlNets for new backbones can
be a significant burden for many users. Furthermore, applying ControlNets
independently to different frames cannot effectively maintain object temporal
consistency. To address these challenges, we introduce Ctrl-Adapter, an
efficient and versatile framework that adds diverse controls to any image/video
diffusion model through the adaptation of pretrained ControlNets. Ctrl-Adapter
offers strong and diverse capabilities, including image and video control,
sparse-frame video control, fine-grained patch-level multi-condition control
(via an MoE router), zero-shot adaptation to unseen conditions, and supports a
variety of downstream tasks beyond spatial control, including video editing,
video style transfer, and text-guided motion control. With six diverse
U-Net/DiT-based image/video diffusion models (SDXL, PixArt-$\alpha$, I2VGen-XL,
SVD, Latte, Hotshot-XL), Ctrl-Adapter matches the performance of pretrained
ControlNets on COCO and achieves the state-of-the-art on DAVIS 2017 with
significantly lower computation (< 10 GPU hours).
