  Sharpness-Aware Minimization (SAM) is an optimizer that takes a descent step
based on the gradient at a perturbation $y_t = x_t + \rho \frac{\nabla
f(x_t)}{\lVert \nabla f(x_t) \rVert}$ of the current point $x_t$. Existing
studies prove convergence of SAM for smooth functions, but they do so by
assuming decaying perturbation size $\rho$ and/or no gradient normalization in
$y_t$, which is detached from practice. To address this gap, we study
deterministic/stochastic versions of SAM with practical configurations (i.e.,
constant $\rho$ and gradient normalization in $y_t$) and explore their
convergence properties on smooth functions with (non)convexity assumptions.
Perhaps surprisingly, in many scenarios, we find out that SAM has limited
capability to converge to global minima or stationary points. For smooth
strongly convex functions, we show that while deterministic SAM enjoys tight
global convergence rates of $\tilde \Theta(\frac{1}{T^2})$, the convergence
bound of stochastic SAM suffers an inevitable additive term $O(\rho^2)$,
indicating convergence only up to neighborhoods of optima. In fact, such
$O(\rho^2)$ factors arise for stochastic SAM in all the settings we consider,
and also for deterministic SAM in nonconvex cases; importantly, we prove by
examples that such terms are unavoidable. Our results highlight vastly
different characteristics of SAM with vs. without decaying perturbation size or
gradient normalization, and suggest that the intuitions gained from one version
may not apply to the other.
