  We propose new limiting dynamics for stochastic gradient descent in the small
learning rate regime called stochastic modified flows. These SDEs are driven by
a cylindrical Brownian motion and improve the so-called stochastic modified
equations by having regular diffusion coefficients and by matching the
multi-point statistics. As a second contribution, we introduce distribution
dependent stochastic modified flows which we prove to describe the fluctuating
limiting dynamics of stochastic gradient descent in the small learning rate -
infinite width scaling regime.
