  We propose a model that is able to perform unsupervised physical parameter
estimation of systems from video, where the differential equations governing
the scene dynamics are known, but labeled states or objects are not available.
Existing physical scene understanding methods require either object state
supervision, or do not integrate with differentiable physics to learn
interpretable system parameters and states. We address this problem through a
physics-as-inverse-graphics approach that brings together
vision-as-inverse-graphics and differentiable physics engines, enabling objects
and explicit state and velocity representations to be discovered. This
framework allows us to perform long term extrapolative video prediction, as
well as vision-based model-predictive control. Our approach significantly
outperforms related unsupervised methods in long-term future frame prediction
of systems with interacting objects (such as ball-spring or 3-body
gravitational systems), due to its ability to build dynamics into the model as
an inductive bias. We further show the value of this tight vision-physics
integration by demonstrating data-efficient learning of vision-actuated
model-based control for a pendulum system. We also show that the controller's
interpretability provides unique capabilities in goal-driven control and
physical reasoning for zero-data adaptation.
