  In this paper, we aim to model 3D scene dynamics from multi-view videos.
Unlike the majority of existing works which usually focus on the common task of
novel view synthesis within the training time period, we propose to
simultaneously learn the geometry, appearance, and physical velocity of 3D
scenes only from video frames, such that multiple desirable applications can be
supported, including future frame extrapolation, unsupervised 3D semantic scene
decomposition, and dynamic motion transfer. Our method consists of three major
components, 1) the keyframe dynamic radiance field, 2) the interframe velocity
field, and 3) a joint keyframe and interframe optimization module which is the
core of our framework to effectively train both networks. To validate our
method, we further introduce two dynamic 3D datasets: 1) Dynamic Object
dataset, and 2) Dynamic Indoor Scene dataset. We conduct extensive experiments
on multiple datasets, demonstrating the superior performance of our method over
all baselines, particularly in the critical tasks of future frame extrapolation
and unsupervised 3D semantic scene decomposition.
