  From just a glance, humans can make rich predictions about the future state
of a wide range of physical systems. On the other hand, modern approaches from
engineering, robotics, and graphics are often restricted to narrow domains and
require direct measurements of the underlying states. We introduce the Visual
Interaction Network, a general-purpose model for learning the dynamics of a
physical system from raw visual observations. Our model consists of a
perceptual front-end based on convolutional neural networks and a dynamics
predictor based on interaction networks. Through joint training, the perceptual
front-end learns to parse a dynamic visual scene into a set of factored latent
object representations. The dynamics predictor learns to roll these states
forward in time by computing their interactions and dynamics, producing a
predicted physical trajectory of arbitrary length. We found that from just six
input video frames the Visual Interaction Network can generate accurate future
trajectories of hundreds of time steps on a wide range of physical systems. Our
model can also be applied to scenes with invisible objects, inferring their
future states from their effects on the visible objects, and can implicitly
infer the unknown mass of objects. Our results demonstrate that the perceptual
module and the object-based dynamics predictor module can induce factored
latent representations that support accurate dynamical predictions. This work
opens new opportunities for model-based decision-making and planning from raw
sensory observations in complex physical environments.
