  Understanding how neural networks learn remains one of the central challenges
in machine learning research. From random at the start of training, the weights
of a neural network evolve in such a way as to be able to perform a variety of
tasks, like classifying images. Here we study the emergence of structure in the
weights by applying methods from topological data analysis. We train simple
feedforward neural networks on the MNIST dataset and monitor the evolution of
the weights. When initialized to zero, the weights follow trajectories that
branch off recurrently, thus generating trees that describe the growth of the
effective capacity of each layer. When initialized to tiny random values, the
weights evolve smoothly along two-dimensional surfaces. We show that natural
coordinates on these learning surfaces correspond to important factors of
variation.
