  From a simplified analysis of adaptive methods, we derive AvaGrad, a new
optimizer which outperforms SGD on vision tasks when its adaptability is
properly tuned. We observe that the power of our method is partially explained
by a decoupling of learning rate and adaptability, greatly simplifying
hyperparameter search. In light of this observation, we demonstrate that,
against conventional wisdom, Adam can also outperform SGD on vision tasks, as
long as the coupling between its learning rate and adaptability is taken into
account. In practice, AvaGrad matches the best results, as measured by
generalization accuracy, delivered by any existing optimizer (SGD or adaptive)
across image classification (CIFAR, ImageNet) and character-level language
modelling (Penn Treebank) tasks.
