  We establish a sharp uniform-in-time error estimate for the Stochastic
Gradient Langevin Dynamics (SGLD), which is a popular sampling algorithm. Under
mild assumptions, we obtain a uniform-in-time $O(\eta^2)$ bound for the
KL-divergence between the SGLD iteration and the Langevin diffusion, where
$\eta$ is the step size (or learning rate). Our analysis is also valid for
varying step sizes. Based on this, we are able to obtain an $O(\eta)$ bound for
the distance between the SGLD iteration and the invariant distribution of the
Langevin diffusion, in terms of Wasserstein or total variation distances.
