  We introduce the Singular Value Representation (SVR), a new method to
represent the internal state of neural networks using SVD factorization of the
weights. This construction yields a new weighted graph connecting what we call
spectral neurons, that correspond to specific activation patterns of classical
neurons. We derive a precise statistical framework to discriminate meaningful
connections between spectral neurons for fully connected and convolutional
layers.
  To demonstrate the usefulness of our approach for machine learning research,
we highlight two discoveries we made using the SVR. First, we highlight the
emergence of a dominant connection in VGG networks that spans multiple deep
layers. Second, we witness, without relying on any input data, that batch
normalization can induce significant connections between near-kernels of deep
layers, leading to a remarkable spontaneous sparsification phenomenon.
