  We develop a framework for analyzing the training and learning rate dynamics
on a large class of high-dimensional optimization problems, which we call the
high line, trained using one-pass stochastic gradient descent (SGD) with
adaptive learning rates. We give exact expressions for the risk and learning
rate curves in terms of a deterministic solution to a system of ODEs. We then
investigate in detail two adaptive learning rates -- an idealized exact line
search and AdaGrad-Norm -- on the least squares problem. When the data
covariance matrix has strictly positive eigenvalues, this idealized exact line
search strategy can exhibit arbitrarily slower convergence when compared to the
optimal fixed learning rate with SGD. Moreover we exactly characterize the
limiting learning rate (as time goes to infinity) for line search in the
setting where the data covariance has only two distinct eigenvalues. For
noiseless targets, we further demonstrate that the AdaGrad-Norm learning rate
converges to a deterministic constant inversely proportional to the average
eigenvalue of the data covariance matrix, and identify a phase transition when
the covariance density of eigenvalues follows a power law distribution.
