  Recently, flat-minima optimizers, which seek to find parameters in low-loss
neighborhoods, have been shown to improve a neural network's generalization
performance over stochastic and adaptive gradient-based optimizers. Two methods
have received significant attention due to their scalability: 1. Stochastic
Weight Averaging (SWA), and 2. Sharpness-Aware Minimization (SAM). However,
there has been limited investigation into their properties and no systematic
benchmarking of them across different domains. We fill this gap here by
comparing the loss surfaces of the models trained with each method and through
broad benchmarking across computer vision, natural language processing, and
graph representation learning tasks. We discover several surprising findings
from these results, which we hope will help researchers further improve deep
learning optimizers, and practitioners identify the right optimizer for their
problem.
