  A natural approach to generative modeling of videos is to represent them as a
composition of moving objects. Recent works model a set of 2D sprites over a
slowly-varying background, but without considering the underlying 3D scene that
gives rise to them. We instead propose to model a video as the view seen while
moving through a scene with multiple 3D objects and a 3D background. Our model
is trained from monocular videos without any supervision, yet learns to
generate coherent 3D scenes containing several moving objects. We conduct
detailed experiments on two datasets, going beyond the visual complexity
supported by state-of-the-art generative approaches. We evaluate our method on
depth-prediction and 3D object detection -- tasks which cannot be addressed by
those earlier works -- and show it out-performs them even on 2D instance
segmentation and tracking.
