  In this work, we study the implications of the implicit bias of gradient flow
on generalization and adversarial robustness in ReLU networks. We focus on a
setting where the data consists of clusters and the correlations between
cluster means are small, and show that in two-layer ReLU networks gradient flow
is biased towards solutions that generalize well, but are highly vulnerable to
adversarial examples. Our results hold even in cases where the network has many
more parameters than training examples. Despite the potential for harmful
overfitting in such overparameterized settings, we prove that the implicit bias
of gradient flow prevents it. However, the implicit bias also leads to
non-robust solutions (susceptible to small adversarial $\ell_2$-perturbations),
even though robust networks that fit the data exist.
