  We analyze the input-output behavior of residual networks from a dynamical
system point of view by disentangling the residual dynamics from the output
activities before the classification stage. For a network with simple skip
connections between every successive layer, and for logistic activation
function, and shared weights between layers, we show analytically that there is
a cooperation and competition dynamics between residuals corresponding to each
input dimension. Interpreting these kind of networks as nonlinear filters, the
steady state value of the residuals in the case of attractor networks are
indicative of the common features between different input dimensions that the
network has observed during training, and has encoded in those components. In
cases where residuals do not converge to an attractor state, their internal
dynamics are separable for each input class, and the network can reliably
approximate the output. We bring analytical and empirical evidence that
residual networks classify inputs based on the integration of the transient
dynamics of the residuals, and will show how the network responds to input
perturbations. We compare the network dynamics for a ResNet and a Multi-Layer
Perceptron and show that the internal dynamics, and the noise evolution are
fundamentally different in these networks, and ResNets are more robust to noisy
inputs. Based on these findings, we also develop a new method to adjust the
depth for residual networks during training. As it turns out, after pruning the
depth of a ResNet using this algorithm, the network is still capable of
classifying inputs with a high accuracy.
