  Neural network training is inherently sequential where the layers finish the
forward propagation in succession, followed by the calculation and
back-propagation of gradients (based on a loss function) starting from the last
layer. The sequential computations significantly slow down neural network
training, especially the deeper ones. Prediction has been successfully used in
many areas of computer architecture to speed up sequential processing.
Therefore, we propose ADA-GP, which uses gradient prediction adaptively to
speed up deep neural network (DNN) training while maintaining accuracy. ADA-GP
works by incorporating a small neural network to predict gradients for
different layers of a DNN model. ADA-GP uses a novel tensor reorganization
method to make it feasible to predict a large number of gradients. ADA-GP
alternates between DNN training using backpropagated gradients and DNN training
using predicted gradients. ADA-GP adaptively adjusts when and for how long
gradient prediction is used to strike a balance between accuracy and
performance. Last but not least, we provide a detailed hardware extension in a
typical DNN accelerator to realize the speed up potential from gradient
prediction. Our extensive experiments with fifteen DNN models show that ADA-GP
can achieve an average speed up of 1.47X with similar or even higher accuracy
than the baseline models. Moreover, it consumes, on average, 34% less energy
due to reduced off-chip memory accesses compared to the baseline accelerator.
