  Models recently used in the literature proving residual networks (ResNets)
are better than linear predictors are actually different from standard ResNets
that have been widely used in computer vision. In addition to the assumptions
such as scalar-valued output or single residual block, these models have no
nonlinearities at the final residual representation that feeds into the final
affine layer. To codify such a difference in nonlinearities and reveal a linear
estimation property, we define ResNEsts, i.e., Residual Nonlinear Estimators,
by simply dropping nonlinearities at the last residual representation from
standard ResNets. We show that wide ResNEsts with bottleneck blocks can always
guarantee a very desirable training property that standard ResNets aim to
achieve, i.e., adding more blocks does not decrease performance given the same
set of basis elements. To prove that, we first recognize ResNEsts are basis
function models that are limited by a coupling problem in basis learning and
linear prediction. Then, to decouple prediction weights from basis learning, we
construct a special architecture termed augmented ResNEst (A-ResNEst) that
always guarantees no worse performance with the addition of a block. As a
result, such an A-ResNEst establishes empirical risk lower bounds for a ResNEst
using corresponding bases. Our results demonstrate ResNEsts indeed have a
problem of diminishing feature reuse; however, it can be avoided by
sufficiently expanding or widening the input space, leading to the
above-mentioned desirable property. Inspired by the DenseNets that have been
shown to outperform ResNets, we also propose a corresponding new model called
Densely connected Nonlinear Estimator (DenseNEst). We show that any DenseNEst
can be represented as a wide ResNEst with bottleneck blocks. Unlike ResNEsts,
DenseNEsts exhibit the desirable property without any special architectural
re-design.
