  In this work, we offer a theoretical analysis of two modern optimization
techniques for training large and complex models: (i) adaptive optimization
algorithms, such as Adam, and (ii) the model exponential moving average (EMA).
Specifically, we demonstrate that a clipped version of Adam with model EMA
achieves the optimal convergence rates in various nonconvex optimization
settings, both smooth and nonsmooth. Moreover, when the scale varies
significantly across different coordinates, we demonstrate that the
coordinate-wise adaptivity of Adam is provably advantageous. Notably, unlike
previous analyses of Adam, our analysis crucially relies on its core elements
-- momentum and discounting factors -- as well as model EMA, motivating their
wide applications in practice.
