  We prove that, for the fundamental regression task of learning a single
neuron, training a one-hidden layer ReLU network of any width by gradient flow
from a small initialisation converges to zero loss and is implicitly biased to
minimise the rank of network parameters. By assuming that the training points
are correlated with the teacher neuron, we complement previous work that
considered orthogonal datasets. Our results are based on a detailed
non-asymptotic analysis of the dynamics of each hidden neuron throughout the
training. We also show and characterise a surprising distinction in this
setting between interpolator networks of minimal rank and those of minimal
Euclidean norm. Finally we perform a range of numerical experiments, which
corroborate our theoretical findings.
