  In machine learning, there is a long history of trying to build neural
networks that can learn from fewer example data by baking in strong geometric
priors. However, it is not always clear a priori what geometric constraints are
appropriate for a given task. Here, we consider the possibility that one can
uncover useful geometric inductive biases by studying how training molds the
Riemannian geometry induced by unconstrained neural network feature maps. We
first show that at infinite width, neural networks with random parameters
induce highly symmetric metrics on input space. This symmetry is broken by
feature learning: networks trained to perform classification tasks learn to
magnify local areas along decision boundaries. This holds in deep networks
trained on high-dimensional image classification tasks, and even in
self-supervised representation learning. These results begins to elucidate how
training shapes the geometry induced by unconstrained neural network feature
maps, laying the groundwork for an understanding of this richly nonlinear form
of feature learning.
