  Understanding implicit bias of gradient descent for generalization capability
of ReLU networks has been an important research topic in machine learning
research. Unfortunately, even for a single ReLU neuron trained with the square
loss, it was recently shown impossible to characterize the implicit
regularization in terms of a norm of model parameters (Vardi & Shamir, 2021).
In order to close the gap toward understanding intriguing generalization
behavior of ReLU networks, here we examine the gradient flow dynamics in the
parameter space when training single-neuron ReLU networks. Specifically, we
discover an implicit bias in terms of support vectors, which plays a key role
in why and how ReLU networks generalize well. Moreover, we analyze gradient
flows with respect to the magnitude of the norm of initialization, and show
that the norm of the learned weight strictly increases through the gradient
flow. Lastly, we prove the global convergence of single ReLU neuron for $d = 2$
case.
