  We give a description of the high-dimensional limit of one-pass single-batch
stochastic gradient descent (SGD) on a least squares problem. This limit is
taken with non-vanishing step-size, and with proportionally related number of
samples to problem-dimensionality. The limit is described in terms of a
stochastic differential equation in high dimensions, which is shown to
approximate the state evolution of SGD. As a corollary, the statistical risk is
shown to be approximated by the solution of a convolution-type Volterra
equation with vanishing errors as dimensionality tends to infinity. The sense
of convergence is the weakest that shows that statistical risks of the two
processes coincide. This is distinguished from existing analyses by the type of
high-dimensional limit given as well as generality of the covariance structure
of the samples.
