  Deep neural networks often suffer from poor generalization due to complex and
non-convex loss landscapes. Sharpness-Aware Minimization (SAM) is a popular
solution that smooths the loss landscape by minimizing the maximized change of
training loss when adding a perturbation to the weight. However, indiscriminate
perturbation of SAM on all parameters is suboptimal and results in excessive
computation, double the overhead of common optimizers like Stochastic Gradient
Descent (SGD). In this paper, we propose Sparse SAM (SSAM), an efficient and
effective training scheme that achieves sparse perturbation by a binary mask.
To obtain the sparse mask, we provide two solutions based on Fisher information
and dynamic sparse training, respectively. We investigate the impact of
different masks, including unstructured, structured, and $N$:$M$ structured
patterns, as well as explicit and implicit forms of implementing sparse
perturbation. We theoretically prove that SSAM can converge at the same rate as
SAM, i.e., $O(\log T/\sqrt{T})$. Sparse SAM has the potential to accelerate
training and smooth the loss landscape effectively. Extensive experimental
results on CIFAR and ImageNet-1K confirm that our method is superior to SAM in
terms of efficiency, and the performance is preserved or even improved with a
perturbation of merely 50\% sparsity. Code is available at
https://github.com/Mi-Peng/Systematic-Investigation-of-Sparse-Perturbed-Sharpness-Aware-Minimization-Optimizer.
