  A recent breakthrough in deep learning theory shows that the training of
over-parameterized deep neural networks can be characterized by a kernel
function called \textit{neural tangent kernel} (NTK). However, it is known that
this type of results does not perfectly match the practice, as NTK-based
analysis requires the network weights to stay very close to their
initialization throughout training, and cannot handle regularizers or gradient
noises. In this paper, we provide a generalized neural tangent kernel analysis
and show that noisy gradient descent with weight decay can still exhibit a
"kernel-like" behavior. This implies that the training loss converges linearly
up to a certain accuracy. We also establish a novel generalization error bound
for two-layer neural networks trained by noisy gradient descent with weight
decay.
