  Several variants of stochastic gradient descent (SGD) have been proposed to
improve the learning effectiveness and efficiency when training deep neural
networks, among which some recent influential attempts would like to adaptively
control the parameter-wise learning rate (e.g., Adam and RMSProp). Although
they show a large improvement in convergence speed, most adaptive learning rate
methods suffer from compromised generalization compared with SGD. In this
paper, we proposed an Adaptive Gradient Method with Resilience and Momentum
(AdaRem), motivated by the observation that the oscillations of network
parameters slow the training, and give a theoretical proof of convergence. For
each parameter, AdaRem adjusts the parameter-wise learning rate according to
whether the direction of one parameter changes in the past is aligned with the
direction of the current gradient, and thus encourages long-term consistent
parameter updating with much fewer oscillations. Comprehensive experiments have
been conducted to verify the effectiveness of AdaRem when training various
models on a large-scale image recognition dataset, e.g., ImageNet, which also
demonstrate that our method outperforms previous adaptive learning rate-based
algorithms in terms of the training speed and the test error, respectively.
