  This paper proposes an efficient optimizer called AdaPlus which integrates
Nesterov momentum and precise stepsize adjustment on AdamW basis. AdaPlus
combines the advantages of AdamW, Nadam, and AdaBelief and, in particular, does
not introduce any extra hyper-parameters. We perform extensive experimental
evaluations on three machine learning tasks to validate the effectiveness of
AdaPlus. The experiment results validate that AdaPlus (i) among all the
evaluated adaptive methods, performs most comparable with (even slightly better
than) SGD with momentum on image classification tasks and (ii) outperforms
other state-of-the-art optimizers on language modeling tasks and illustrates
pretty high stability when training GANs. The experiment code of AdaPlus will
be accessible at: https://github.com/guanleics/AdaPlus.
