  In standard neural network training, the gradients in the backward pass are
determined by the forward pass. As a result, the two stages are coupled. This
is how most neural networks are trained currently. However, gradient
modification in the backward pass has seldom been studied in the literature. In
this paper we explore decoupled training, where we alter the gradients in the
backward pass. We propose a simple yet powerful method called PowerGrad
Transform, that alters the gradients before the weight update in the backward
pass and significantly enhances the predictive performance of the neural
network. PowerGrad Transform trains the network to arrive at a better optima at
convergence. It is computationally extremely efficient, virtually adding no
additional cost to either memory or compute, but results in improved final
accuracies on both the training and test sets. PowerGrad Transform is easy to
integrate into existing training routines, requiring just a few lines of code.
PowerGrad Transform accelerates training and makes it possible for the network
to better fit the training data. With decoupled training, PowerGrad Transform
improves baseline accuracies for ResNet-50 by 0.73%, for SE-ResNet-50 by 0.66%
and by more than 1.0% for the non-normalized ResNet-18 network on the ImageNet
classification task.
