  3D scene representation for robot manipulation should capture three key
object properties: permanency -- objects that become occluded over time
continue to exist; amodal completeness -- objects have 3D occupancy, even if
only partial observations are available; spatiotemporal continuity -- the
movement of each object is continuous over space and time. In this paper, we
introduce 3D Dynamic Scene Representation (DSR), a 3D volumetric scene
representation that simultaneously discovers, tracks, reconstructs objects, and
predicts their dynamics while capturing all three properties. We further
propose DSR-Net, which learns to aggregate visual observations over multiple
interactions to gradually build and refine DSR. Our model achieves
state-of-the-art performance in modeling 3D scene dynamics with DSR on both
simulated and real data. Combined with model predictive control, DSR-Net
enables accurate planning in downstream robotic manipulation tasks such as
planar pushing. Video is available at https://youtu.be/GQjYG3nQJ80.
