  Generating videos with realistic and physically plausible motion is one of
the main recent challenges in computer vision. While diffusion models are
achieving compelling results in image generation, video diffusion models are
limited by heavy training and huge models, resulting in videos that are still
biased to the training dataset. In this work we propose MotionCraft, a new
zero-shot video generator to craft physics-based and realistic videos.
MotionCraft is able to warp the noise latent space of an image diffusion model,
such as Stable Diffusion, by applying an optical flow derived from a physics
simulation. We show that warping the noise latent space results in coherent
application of the desired motion while allowing the model to generate missing
elements consistent with the scene evolution, which would otherwise result in
artefacts or missing content if the flow was applied in the pixel space. We
compare our method with the state-of-the-art Text2Video-Zero reporting
qualitative and quantitative improvements, demonstrating the effectiveness of
our approach to generate videos with finely-prescribed complex motion dynamics.
Project page: https://mezzelfo.github.io/MotionCraft/
