  View-predictive generative models provide strong priors for lifting
object-centric images and videos into 3D and 4D through rendering and score
distillation objectives. A question then remains: what about lifting complete
multi-object dynamic scenes? There are two challenges in this direction: First,
rendering error gradients are often insufficient to recover fast object motion,
and second, view predictive generative models work much better for objects than
whole scenes, so, score distillation objectives cannot currently be applied at
the scene level directly. We present DreamScene4D, the first approach to
generate 3D dynamic scenes of multiple objects from monocular videos via
360-degree novel view synthesis. Our key insight is a "decompose-recompose"
approach that factorizes the video scene into the background and object tracks,
while also factorizing object motion into 3 components: object-centric
deformation, object-to-world-frame transformation, and camera motion. Such
decomposition permits rendering error gradients and object view-predictive
models to recover object 3D completions and deformations while bounding box
tracks guide the large object movements in the scene. We show extensive results
on challenging DAVIS, Kubric, and self-captured videos with quantitative
comparisons and a user preference study. Besides 4D scene generation,
DreamScene4D obtains accurate 2D persistent point track by projecting the
inferred 3D trajectories to 2D. We will release our code and hope our work will
stimulate more research on fine-grained 4D understanding from videos.
