  Learning dynamics is at the heart of many important applications of machine
learning (ML), such as robotics and autonomous driving. In these settings, ML
algorithms typically need to reason about a physical system using high
dimensional observations, such as images, without access to the underlying
state. Recently, several methods have proposed to integrate priors from
classical mechanics into ML models to address the challenge of physical
reasoning from images. In this work, we take a sober look at the current
capabilities of these models. To this end, we introduce a suite consisting of
17 datasets with visual observations based on physical systems exhibiting a
wide range of dynamics. We conduct a thorough and detailed comparison of the
major classes of physically inspired methods alongside several strong
baselines. While models that incorporate physical priors can often learn latent
spaces with desirable properties, our results demonstrate that these methods
fail to significantly improve upon standard techniques. Nonetheless, we find
that the use of continuous and time-reversible dynamics benefits models of all
classes.
