  A longstanding debate surrounds the related hypotheses that low-curvature
minima generalize better, and that SGD discourages curvature. We offer a more
complete and nuanced view in support of both. First, we show that curvature
harms test performance through two new mechanisms, the shift-curvature and
bias-curvature, in addition to a known parameter-covariance mechanism. The
three curvature-mediated contributions to test performance are
reparametrization-invariant although curvature is not. The shift in the
shift-curvature is the line connecting train and test local minima, which
differ due to dataset sampling or distribution shift. Although the shift is
unknown at training time, the shift-curvature can still be mitigated by
minimizing overall curvature. Second, we derive a new, explicit SGD
steady-state distribution showing that SGD optimizes an effective potential
related to but different from train loss, and that SGD noise mediates a
trade-off between deep versus low-curvature regions of this effective
potential. Third, combining our test performance analysis with the SGD steady
state shows that for small SGD noise, the shift-curvature may be the most
significant of the three mechanisms. Our experiments confirm the impact of
shift-curvature on test loss, and further explore the relationship between SGD
noise and curvature.
