  We study universal traits which emerge both in real-world complex datasets,
as well as in artificially generated ones. Our approach is to analogize data to
a physical system and employ tools from statistical physics and Random Matrix
Theory (RMT) to reveal their underlying structure. We focus on the
feature-feature covariance matrix, analyzing both its local and global
eigenvalue statistics. Our main observations are: (i) The power-law scalings
that the bulk of its eigenvalues exhibit are vastly different for uncorrelated
normally distributed data compared to real-world data, (ii) this scaling
behavior can be completely modeled by generating Gaussian data with long range
correlations, (iii) both generated and real-world datasets lie in the same
universality class from the RMT perspective, as chaotic rather than integrable
systems, (iv) the expected RMT statistical behavior already manifests for
empirical covariance matrices at dataset sizes significantly smaller than those
conventionally used for real-world training, and can be related to the number
of samples required to approximate the population power-law scaling behavior,
(v) the Shannon entropy is correlated with local RMT structure and eigenvalues
scaling, is substantially smaller in strongly correlated datasets compared to
uncorrelated ones, and requires fewer samples to reach the distribution
entropy. These findings show that with sufficient sample size, the Gram matrix
of natural image datasets can be well approximated by a Wishart random matrix
with a simple covariance structure, opening the door to rigorous studies of
neural network dynamics and generalization which rely on the data Gram matrix.
