  All physical laws are described as relationships between state variables that
give a complete and non-redundant description of the relevant system dynamics.
However, despite the prevalence of computing power and AI, the process of
identifying the hidden state variables themselves has resisted automation. Most
data-driven methods for modeling physical phenomena still assume that observed
data streams already correspond to relevant state variables. A key challenge is
to identify the possible sets of state variables from scratch, given only
high-dimensional observational data. Here we propose a new principle for
determining how many state variables an observed system is likely to have, and
what these variables might be, directly from video streams. We demonstrate the
effectiveness of this approach using video recordings of a variety of physical
dynamical systems, ranging from elastic double pendulums to fire flames.
Without any prior knowledge of the underlying physics, our algorithm discovers
the intrinsic dimension of the observed dynamics and identifies candidate sets
of state variables. We suggest that this approach could help catalyze the
understanding, prediction and control of increasingly complex systems. Project
website is at: https://www.cs.columbia.edu/~bchen/neural-state-variables
