  Recently, diffusion models like StableDiffusion have achieved impressive
image generation results. However, the generation process of such diffusion
models is uncontrollable, which makes it hard to generate videos with
continuous and consistent content. In this work, by using the diffusion model
with ControlNet, we proposed a new motion-guided video-to-video translation
framework called VideoControlNet to generate various videos based on the given
prompts and the condition from the input video. Inspired by the video codecs
that use motion information for reducing temporal redundancy, our framework
uses motion information to prevent the regeneration of the redundant areas for
content consistency. Specifically, we generate the first frame (i.e., the
I-frame) by using the diffusion model with ControlNet. Then we generate other
key frames (i.e., the P-frame) based on the previous I/P-frame by using our
newly proposed motion-guided P-frame generation (MgPG) method, in which the
P-frames are generated based on the motion information and the occlusion areas
are inpainted by using the diffusion model. Finally, the rest frames (i.e., the
B-frame) are generated by using our motion-guided B-frame interpolation (MgBI)
module. Our experiments demonstrate that our proposed VideoControlNet inherits
the generation capability of the pre-trained large diffusion model and extends
the image diffusion model to the video diffusion model by using motion
information. More results are provided at our project page.
