  Modern deep learning models are over-parameterized, where the optimization
setup strongly affects the generalization performance. A key element of
reliable optimization for these systems is the modification of the loss
function. Sharpness-Aware Minimization (SAM) modifies the underlying loss
function to guide descent methods towards flatter minima, which arguably have
better generalization abilities. In this paper, we focus on a variant of SAM
known as mSAM, which, during training, averages the updates generated by
adversarial perturbations across several disjoint shards of a mini-batch.
Recent work suggests that mSAM can outperform SAM in terms of test accuracy.
However, a comprehensive empirical study of mSAM is missing from the literature
-- previous results have mostly been limited to specific architectures and
datasets. To that end, this paper presents a thorough empirical evaluation of
mSAM on various tasks and datasets. We provide a flexible implementation of
mSAM and compare the generalization performance of mSAM to the performance of
SAM and vanilla training on different image classification and natural language
processing tasks. We also conduct careful experiments to understand the
computational cost of training with mSAM, its sensitivity to hyperparameters
and its correlation with the flatness of the loss landscape. Our analysis
reveals that mSAM yields superior generalization performance and flatter
minima, compared to SAM, across a wide range of tasks without significantly
increasing computational costs.
