  Aided by text-to-image and text-to-video diffusion models, existing 4D
content creation pipelines utilize score distillation sampling to optimize the
entire dynamic 3D scene. However, as these pipelines generate 4D content from
text or image inputs, they incur significant time and effort in prompt
engineering through trial and error. This work introduces 4DGen, a novel,
holistic framework for grounded 4D content creation that decomposes the 4D
generation task into multiple stages. We identify static 3D assets and
monocular video sequences as key components in constructing the 4D content. Our
pipeline facilitates conditional 4D generation, enabling users to specify
geometry (3D assets) and motion (monocular videos), thus offering superior
control over content creation. Furthermore, we construct our 4D representation
using dynamic 3D Gaussians, which permits efficient, high-resolution
supervision through rendering during training, thereby facilitating
high-quality 4D generation. Additionally, we employ spatial-temporal pseudo
labels on anchor frames, along with seamless consistency priors implemented
through 3D-aware score distillation sampling and smoothness regularizations.
Compared to existing baselines, our approach yields competitive results in
faithfully reconstructing input signals and realistically inferring renderings
from novel viewpoints and timesteps. Most importantly, our method supports
grounded generation, offering users enhanced control, a feature difficult to
achieve with previous methods. Project page:
https://vita-group.github.io/4DGen/
