  Recent advancements in 3D generation are predominantly propelled by
improvements in 3D-aware image diffusion models which are pretrained on
Internet-scale image data and fine-tuned on massive 3D data, offering the
capability of producing highly consistent multi-view images. However, due to
the scarcity of synchronized multi-view video data, it is impractical to adapt
this paradigm to 4D generation directly. Despite that, the available video and
3D data are adequate for training video and multi-view diffusion models
separately that can provide satisfactory dynamic and geometric priors
respectively. To take advantage of both, this paper present Diffusion$^2$, a
novel framework for dynamic 3D content creation that reconciles the knowledge
about geometric consistency and temporal smoothness from these models to
directly sample dense multi-view multi-frame images which can be employed to
optimize continuous 4D representation. Specifically, we design a simple yet
effective denoising strategy via score composition of pretrained video and
multi-view diffusion models based on the probability structure of the target
image array. Owing to the high parallelism of the proposed image generation
process and the efficiency of the modern 4D reconstruction pipeline, our
framework can generate 4D content within few minutes. Additionally, our method
circumvents the reliance on 4D data, thereby having the potential to benefit
from the scaling of the foundation video and multi-view diffusion models.
Extensive experiments demonstrate the efficacy of our proposed framework and
its ability to flexibly handle various types of prompts.
