  The residual neural network (ResNet) is a popular deep network architecture
which has the ability to obtain high-accuracy results on several image
processing problems. In order to analyze the behavior and structure of ResNet,
recent work has been on establishing connections between ResNets and
continuous-time optimal control problems. In this work, we show that the
post-activation ResNet is related to an optimal control problem with
differential inclusions, and provide continuous-time stability results for the
differential inclusion associated with ResNet. Motivated by the stability
conditions, we show that alterations of either the architecture or the
optimization problem can generate variants of ResNet which improve the
theoretical stability bounds. In addition, we establish stability bounds for
the full (discrete) network associated with two variants of ResNet, in
particular, bounds on the growth of the features and a measure of the
sensitivity of the features with respect to perturbations. These results also
help to show the relationship between the depth, regularization, and stability
of the feature space. Computational experiments on the proposed variants show
that the accuracy of ResNet is preserved and that the accuracy seems to be
monotone with respect to the depth and various corruptions.
