  Understanding the geometric properties of gradient descent dynamics is a key
ingredient in deciphering the recent success of very large machine learning
models. A striking observation is that trained over-parameterized models retain
some properties of the optimization initialization. This "implicit bias" is
believed to be responsible for some favorable properties of the trained models
and could explain their good generalization properties. The purpose of this
article is threefold. First, we rigorously expose the definition and basic
properties of "conservation laws", that define quantities conserved during
gradient flows of a given model (e.g. of a ReLU network with a given
architecture) with any training data and any loss. Then we explain how to find
the maximal number of independent conservation laws by performing
finite-dimensional algebraic manipulations on the Lie algebra generated by the
Jacobian of the model. Finally, we provide algorithms to: a) compute a family
of polynomial laws; b) compute the maximal number of (not necessarily
polynomial) independent conservation laws. We provide showcase examples that we
fully work out theoretically. Besides, applying the two algorithms confirms for
a number of ReLU network architectures that all known laws are recovered by the
algorithm, and that there are no other independent laws. Such computational
tools pave the way to understanding desirable properties of optimization
initialization in large machine learning models.
