  We introduce a GP generalization of ResNets (including ResNets as a
particular case). We show that ResNets (and their GP generalization) converge,
in the infinite depth limit, to a generalization of image registration
variational algorithms. Whereas computational anatomy aligns images via warping
of the material space, this generalization aligns ideas (or abstract shapes as
in Plato's theory of forms) via the warping of the RKHS of functions mapping
the input space to the output space. While the Hamiltonian interpretation of
ResNets is not new, it was based on an Ansatz. We do not rely on this Ansatz
and present the first rigorous proof of convergence of ResNets with trained
weights and biases towards a Hamiltonian dynamics driven flow. Our constructive
proof reveals several remarkable properties of ResNets and their GP
generalization. ResNets regressors are kernel regressors with data-dependent
warping kernels. Minimizers of $L_2$ regularized ResNets satisfy a discrete
least action principle implying the near preservation of the norm of weights
and biases across layers. The trained weights of ResNets with $L^2$
regularization can be identified by solving an autonomous Hamiltonian system.
The trained ResNet parameters are unique up to the initial momentum whose
representation is generally sparse. The kernel regularization strategy provides
a provably robust alternative to Dropout for ANNs. We introduce a functional
generalization of GPs leading to error estimates for ResNets. We identify the
(EPDiff) mean fields limit of trained ResNet parameters. We show that the
composition of warping regression blocks with reduced equivariant multichannel
kernels (introduced here) recovers and generalizes CNNs to arbitrary spaces and
groups of transformations.
