  This paper deals with neural networks as dynamical systems governed by
differential or difference equations. It shows that the introduction of skip
connections into network architectures, such as residual networks and dense
networks, turns a system of static equations into a system of dynamical
equations with varying levels of smoothness on the layer-wise transformations.
Closed form solutions for the state space representations of general dense
networks, as well as $k^{th}$ order smooth networks, are found in general
settings. Furthermore, it is shown that imposing $k^{th}$ order smoothness on a
network architecture with $d$-many nodes per layer increases the state space
dimension by a multiple of $k$, and so the effective embedding dimension of the
data manifold is $k \cdot d$-many dimensions. It follows that network
architectures of these types reduce the number of parameters needed to maintain
the same embedding dimension by a factor of $k^2$ when compared to an
equivalent first-order, residual network, significantly motivating the
development of network architectures of these types. Numerical simulations were
run to validate parts of the developed theory.
