  We develop a new formulation of deep learning based on the Mori-Zwanzig (MZ)
formalism of irreversible statistical mechanics. The new formulation is built
upon the well-known duality between deep neural networks and discrete dynamical
systems, and it allows us to directly propagate quantities of interest
(conditional expectations and probability density functions) forward and
backward through the network by means of exact linear operator equations. Such
new equations can be used as a starting point to develop new effective
parameterizations of deep neural networks, and provide a new framework to study
deep-learning via operator theoretic methods. The proposed MZ formulation of
deep learning naturally introduces a new concept, i.e., the memory of the
neural network, which plays a fundamental role in low-dimensional modeling and
parameterization. By using the theory of contraction mappings, we develop
sufficient conditions for the memory of the neural network to decay with the
number of layers. This allows us to rigorously transform deep networks into
shallow ones, e.g., by reducing the number of neurons per layer (using
projection operators), or by reducing the total number of layers (using the
decay property of the memory operator).
