  Despite the non-convex nature of their loss functions, deep neural networks
are known to generalize well when optimized with stochastic gradient descent
(SGD). Recent work conjectures that SGD with proper configuration is able to
find wide and flat local minima, which have been proposed to be associated with
good generalization performance. In this paper, we observe that local minima of
modern deep networks are more than being flat or sharp. Specifically, at a
local minimum there exist many asymmetric directions such that the loss
increases abruptly along one side, and slowly along the opposite side--we
formally define such minima as asymmetric valleys. Under mild assumptions, we
prove that for asymmetric valleys, a solution biased towards the flat side
generalizes better than the exact minimizer. Further, we show that simply
averaging the weights along the SGD trajectory gives rise to such biased
solutions implicitly. This provides a theoretical explanation for the
intriguing phenomenon observed by Izmailov et al. (2018). In addition, we
empirically find that batch normalization (BN) appears to be a major cause for
asymmetric valleys.
