  For many of the physical phenomena around us, we have developed sophisticated
models explaining their behavior. Nevertheless, inferring specifics from visual
observations is challenging due to the high number of causally underlying
physical parameters -- including material properties and external forces. This
paper addresses the problem of inferring such latent physical properties from
observations. Our solution is an iterative refinement procedure with simulation
at its core. The algorithm gradually updates the physical model parameters by
running a simulation of the observed phenomenon and comparing the current
simulation to a real-world observation. The physical similarity is computed
using an embedding function that maps physically similar examples to nearby
points. As a tangible example, we concentrate on flags curling in the wind -- a
seemingly simple phenomenon but physically highly involved. Based on its
underlying physical model and visual manifestation, we propose an instantiation
of the embedding function. For this mapping, modeled as a deep network, we
introduce a spectral decomposition layer that decomposes a video volume into
its temporal spectral power and corresponding frequencies. In experiments, we
demonstrate our method's ability to recover intrinsic and extrinsic physical
parameters from both simulated and real-world video.
