  The generalization mystery of overparametrized deep nets has motivated
efforts to understand how gradient descent (GD) converges to low-loss solutions
that generalize well. Real-life neural networks are initialized from small
random values and trained with cross-entropy loss for classification (unlike
the "lazy" or "NTK" regime of training where analysis was more successful), and
a recent sequence of results (Lyu and Li, 2020; Chizat and Bach, 2020; Ji and
Telgarsky, 2020) provide theoretical evidence that GD may converge to the
"max-margin" solution with zero loss, which presumably generalizes well.
However, the global optimality of margin is proved only in some settings where
neural nets are infinitely or exponentially wide. The current paper is able to
establish this global optimality for two-layer Leaky ReLU nets trained with
gradient flow on linearly separable and symmetric data, regardless of the
width. The analysis also gives some theoretical justification for recent
empirical findings (Kalimeris et al., 2019) on the so-called simplicity bias of
GD towards linear or other "simple" classes of solutions, especially early in
training. On the pessimistic side, the paper suggests that such results are
fragile. A simple data manipulation can make gradient flow converge to a linear
classifier with suboptimal margin.
