  We consider networks, trained via stochastic gradient descent to minimize
$\ell_2$ loss, with the training labels perturbed by independent noise at each
iteration. We characterize the behavior of the training dynamics near any
parameter vector that achieves zero training error, in terms of an implicit
regularization term corresponding to the sum over the data points, of the
squared $\ell_2$ norm of the gradient of the model with respect to the
parameter vector, evaluated at each data point. This holds for networks of any
connectivity, width, depth, and choice of activation function. We interpret
this implicit regularization term for three simple settings: matrix sensing,
two layer ReLU networks trained on one-dimensional data, and two layer networks
with sigmoid activations trained on a single datapoint. For these settings, we
show why this new and general implicit regularization effect drives the
networks towards "simple" models.
