  Training an overparameterized neural network can yield minimizers of
different generalization capabilities despite the same level of training loss.
Meanwhile, with evidence that suggests a strong correlation between the
sharpness of minima and their generalization errors, increasing efforts have
been made to develop optimization methods to explicitly find flat minima as
more generalizable solutions. Despite its contemporary relevance to
overparameterization, however, this sharpness-aware minimization (SAM) strategy
has not been studied much yet as to exactly how it is affected by
overparameterization. Hence, in this work, we analyze SAM under
overparameterization of varying degrees and present both empirical and
theoretical results that indicate a critical influence of overparameterization
on SAM. At first, we conduct extensive numerical experiments across vision,
language, graph, and reinforcement learning domains and show that SAM
consistently improves with overparameterization. Next, we attribute this
phenomenon to the interplay between the enlarged solution space and increased
implicit bias from overparameterization. Further, we prove multiple theoretical
benefits of overparameterization for SAM to attain (i) minima with more uniform
Hessian moments compared to SGD, (ii) much faster convergence at a linear rate,
and (iii) lower test error for two-layer networks. Last but not least, we
discover that the effect of overparameterization is more significantly
pronounced in practical settings of label noise and sparsity, and yet,
sufficient regularization is necessary.
