  Stochastic gradient descent (SGD) has been the dominant optimization method
for training deep neural networks due to its many desirable properties. One of
the more remarkable and least understood quality of SGD is that it generalizes
relatively well on unseen data even when the neural network has millions of
parameters. We hypothesize that in certain cases it is desirable to relax its
intrinsic generalization properties and introduce an extension of SGD called
deep gradient boosting (DGB). The key idea of DGB is that back-propagated
gradients inferred using the chain rule can be viewed as pseudo-residual
targets of a gradient boosting problem. Thus at each layer of a neural network
the weight update is calculated by solving the corresponding boosting problem
using a linear base learner. The resulting weight update formula can also be
viewed as a normalization procedure of the data that arrives at each layer
during the forward pass. When implemented as a separate input normalization
layer (INN) the new architecture shows improved performance on image
recognition tasks when compared to the same architecture without normalization
layers. As opposed to batch normalization (BN), INN has no learnable parameters
however it matches its performance on CIFAR10 and ImageNet classification
tasks.
