  Any representation of data involves arbitrary investigator choices. Because
those choices are external to the data-generating process, each choice leads to
an exact symmetry, corresponding to the group of transformations that takes one
possible representation to another. These are the passive symmetries; they
include coordinate freedom, gauge symmetry, and units covariance, all of which
have led to important results in physics. In machine learning, the most visible
passive symmetry is the relabeling or permutation symmetry of graphs. Our goal
is to understand the implications for machine learning of the many passive
symmetries in play. We discuss dos and don'ts for machine learning practice if
passive symmetries are to be respected. We discuss links to causal modeling,
and argue that the implementation of passive symmetries is particularly
valuable when the goal of the learning problem is to generalize out of sample.
This paper is conceptual: It translates among the languages of physics,
mathematics, and machine-learning. We believe that consideration and
implementation of passive symmetries might help machine learning in the same
ways that it transformed physics in the twentieth century.
