  In the past decade, significant strides in deep learning have led to numerous
groundbreaking applications. Despite these advancements, the understanding of
the high generalizability of deep learning, especially in such an
over-parametrized space, remains limited. Successful applications are often
considered as empirical rather than scientific achievements. For instance, deep
neural networks' (DNNs) internal representations, decision-making mechanism,
absence of overfitting in an over-parametrized space, high generalizability,
etc., remain less understood. This paper delves into the loss landscape of DNNs
through the lens of spin glass in statistical physics, i.e. a system
characterized by a complex energy landscape with numerous metastable states, to
better understand how DNNs work. We investigated a single hidden layer
Rectified Linear Unit (ReLU) neural network model, and introduced several
protocols to examine the analogy between DNNs (trained with datasets including
MNIST and CIFAR10) and spin glass. Specifically, we used (1) random walk in the
parameter space of DNNs to unravel the structures in their loss landscape; (2)
a permutation-interpolation protocol to study the connection between copies of
identical regions in the loss landscape due to the permutation symmetry in the
hidden layers; (3) hierarchical clustering to reveal the hierarchy among
trained solutions of DNNs, reminiscent of the so-called Replica Symmetry
Breaking (RSB) phenomenon (i.e. the Parisi solution) in analogy to spin glass;
(4) finally, we examine the relationship between the degree of the ruggedness
of the loss landscape of the DNN and its generalizability, showing an
improvement of flattened minima.
