  Adaptive optimization methods are well known to achieve superior convergence
relative to vanilla gradient methods. The traditional viewpoint in
optimization, particularly in convex optimization, explains this improved
performance by arguing that, unlike vanilla gradient schemes, adaptive
algorithms mimic the behavior of a second-order method by adapting to the
global geometry of the loss function. We argue that in the context of neural
network optimization, this traditional viewpoint is insufficient. Instead, we
advocate for a local trajectory analysis. For iterate trajectories produced by
running a generic optimization algorithm OPT, we introduce
$R^{\text{OPT}}_{\text{med}}$, a statistic that is analogous to the condition
number of the loss Hessian evaluated at the iterates. Through extensive
experiments, we show that adaptive methods such as Adam bias the trajectories
towards regions where $R^{\text{Adam}}_{\text{med}}$ is small, where one might
expect faster convergence. By contrast, vanilla gradient methods like SGD bias
the trajectories towards regions where $R^{\text{SGD}}_{\text{med}}$ is
comparatively large. We complement these empirical observations with a
theoretical result that provably demonstrates this phenomenon in the simplified
setting of a two-layer linear network. We view our findings as evidence for the
need of a new explanation of the success of adaptive methods, one that is
different than the conventional wisdom.
