  Random label noises (or observational noises) widely exist in practical
machine learning settings. While previous studies primarily focus on the
affects of label noises to the performance of learning, our work intends to
investigate the implicit regularization effects of the label noises, under
mini-batch sampling settings of stochastic gradient descent (SGD), with
assumptions that label noises are unbiased. Specifically, we analyze the
learning dynamics of SGD over the quadratic loss with unbiased label noises,
where we model the dynamics of SGD as a stochastic differentiable equation
(SDE) with two diffusion terms (namely a Doubly Stochastic Model). While the
first diffusion term is caused by mini-batch sampling over the
(label-noiseless) loss gradients as many other works on SGD, our model
investigates the second noise term of SGD dynamics, which is caused by
mini-batch sampling over the label noises, as an implicit regularizer. Our
theoretical analysis finds such implicit regularizer would favor some
convergence points that could stabilize model outputs against perturbation of
parameters (namely inference stability). Though similar phenomenon have been
investigated, our work doesn't assume SGD as an Ornstein-Uhlenbeck like process
and achieve a more generalizable result with convergence of approximation
proved. To validate our analysis, we design two sets of empirical studies to
analyze the implicit regularizer of SGD with unbiased random label noises for
deep neural networks training and linear regression.
