  Neural networks have been used successfully in a variety of fields, which has
led to a great deal of interest in developing a theoretical understanding of
how they store the information needed to perform a particular task. We study
the weight matrices of trained deep neural networks using methods from random
matrix theory (RMT) and show that the statistics of most of the singular values
follow universal RMT predictions. This suggests that they are random and do not
contain system specific information, which we investigate further by comparing
the statistics of eigenvector entries to the universal Porter-Thomas
distribution. We find that for most eigenvectors the hypothesis of randomness
cannot be rejected, and that only eigenvectors belonging to the largest
singular values deviate from the RMT prediction, indicating that they may
encode learned information. In addition, a comparison with RMT predictions also
allows to distinguish networks trained in different learning regimes - from
lazy to rich learning. We analyze the spectral distribution of the large
singular values using the Hill estimator and find that the distribution cannot
in general be characterized by a tail index, i.e. is not of power law type.
