  Gradient regularization, as described in \citet{barrett2021implicit}, is a
highly effective technique for promoting flat minima during gradient descent.
Empirical evidence suggests that this regularization technique can
significantly enhance the robustness of deep learning models against noisy
perturbations, while also reducing test error. In this paper, we explore the
per-example gradient regularization (PEGR) and present a theoretical analysis
that demonstrates its effectiveness in improving both test error and robustness
against noise perturbations. Specifically, we adopt a signal-noise data model
from \citet{cao2022benign} and show that PEGR can learn signals effectively
while suppressing noise. In contrast, standard gradient descent struggles to
distinguish the signal from the noise, leading to suboptimal generalization
performance. Our analysis reveals that PEGR penalizes the variance of pattern
learning, thus effectively suppressing the memorization of noises from the
training data. These findings underscore the importance of variance control in
deep learning training and offer useful insights for developing more effective
training approaches.
