  Understanding the structure of loss landscape of deep neural networks
(DNNs)is obviously important. In this work, we prove an embedding principle
that the loss landscape of a DNN "contains" all the critical points of all the
narrower DNNs. More precisely, we propose a critical embedding such that any
critical point, e.g., local or global minima, of a narrower DNN can be embedded
to a critical point/hyperplane of the target DNN with higher degeneracy and
preserving the DNN output function. The embedding structure of critical points
is independent of loss function and training data, showing a stark difference
from other nonconvex problems such as protein-folding. Empirically, we find
that a wide DNN is often attracted by highly-degenerate critical points that
are embedded from narrow DNNs. The embedding principle provides an explanation
for the general easy optimization of wide DNNs and unravels a potential
implicit low-complexity regularization during the training. Overall, our work
provides a skeleton for the study of loss landscape of DNNs and its
implication, by which a more exact and comprehensive understanding can be
anticipated in the near
