  We consider the fundamental problem of learning linear predictors (i.e.,
separable datasets with zero margin) using neural networks with gradient flow
or gradient descent. Under the assumption of spherically symmetric data
distribution, we show directional convergence guarantees with exact convergence
rate for two-layer non-linear networks with only two hidden nodes, and (deep)
linear networks. Moreover, our discovery is built on dynamic from the
initialization without both initial loss and perfect classification constraint
in contrast to previous works. We also point out and study the challenges in
further strengthening and generalizing our results.
