  This work is substituted by the paper in arXiv:2011.14066.
  Stochastic gradient descent is the de facto algorithm for training deep
neural networks (DNNs). Despite its popularity, it still requires fine tuning
in order to achieve its best performance. This has led to the development of
adaptive methods, that claim automatic hyper-parameter optimization. Recently,
researchers have studied both algorithmic classes via toy examples: e.g., for
over-parameterized linear regression, Wilson et. al. (2017) shows that, while
SGD always converges to the minimum-norm solution, adaptive methods show no
such inclination, leading to worse generalization capabilities. Our aim is to
study this conjecture further. We empirically show that the minimum weight norm
is not necessarily the proper gauge of good generalization in simplified
scenaria, and different models found by adaptive methods could outperform plain
gradient methods. In practical DNN settings, we observe that adaptive methods
can outperform SGD, with larger weight norm output models, but without
necessarily reducing the amount of tuning required.
