  Despite the stunning progress recently in large-scale deep neural network
applications, our understanding of their microstructure, 'energy' functions,
and optimal design remains incomplete. Here, we present a new game-theoretic
framework, called statistical teleodynamics, that reveals important insights
into these key properties. The optimally robust design of such networks
inherently involves computational benefit-cost trade-offs that are not
adequately captured by physics-inspired models. These trade-offs occur as
neurons and connections compete to increase their effective utilities under
resource constraints during training. In a fully trained network, this results
in a state of arbitrage equilibrium, where all neurons in a given layer have
the same effective utility, and all connections to a given layer have the same
effective utility. The equilibrium is characterized by the emergence of two
lognormal distributions of connection weights and neuronal output as the
universal microstructure of large deep neural networks. We call such a network
the Jaynes Machine. Our theoretical predictions are shown to be supported by
empirical data from seven large-scale deep neural networks. We also show that
the Hopfield network and the Boltzmann Machine are the same special case of the
Jaynes Machine.
