  It has long been suggested that the biological brain operates at some
critical point between two different phases, possibly order and chaos. Despite
many indirect empirical evidence from the brain and analytical indication on
simple neural networks, the foundation of this hypothesis on generic non-linear
systems remains unclear. Here we develop a general theory that reveals the
exact edge of chaos is the boundary between the chaotic phase and the
(pseudo)periodic phase arising from Neimark-Sacker bifurcation. This edge is
analytically determined by the asymptotic Jacobian norm values of the
non-linear operator and influenced by the dimensionality of the system. The
optimality at the edge of chaos is associated with the highest information
transfer between input and output at this point similar to that of the logistic
map. As empirical validations, our experiments on the various deep learning
models in computer vision demonstrate the optimality of the models near the
edge of chaos, and we observe that the state-of-art training algorithms push
the models towards such edge as they become more accurate. We further
establishes the theoretical understanding of deep learning model generalization
through asymptotic stability.
