  Recent results in the literature indicate that a residual network (ResNet)
composed of a single residual block outperforms linear predictors, in the sense
that all local minima in its optimization landscape are at least as good as the
best linear predictor. However, these results are limited to a single residual
block (i.e., shallow ResNets), instead of the deep ResNets composed of multiple
residual blocks. We take a step towards extending this result to deep ResNets.
We start by two motivating examples. First, we show that there exist datasets
for which all local minima of a fully-connected ReLU network are no better than
the best linear predictor, whereas a ResNet has strictly better local minima.
Second, we show that even at the global minimum, the representation obtained
from the residual block outputs of a 2-block ResNet do not necessarily improve
monotonically over subsequent blocks, which highlights a fundamental difficulty
in analyzing deep ResNets. Our main theorem on deep ResNets shows under simple
geometric conditions that, any critical point in the optimization landscape is
either (i) at least as good as the best linear predictor; or (ii) the Hessian
at this critical point has a strictly negative eigenvalue. Notably, our theorem
shows that a chain of multiple skip-connections can improve the optimization
landscape, whereas existing results study direct skip-connections to the last
hidden layer or output layer. Finally, we complement our results by showing
benign properties of the "near-identity regions" of deep ResNets, showing
depth-independent upper bounds for the risk attained at critical points as well
as the Rademacher complexity.
