  We provide tight finite-time convergence bounds for gradient descent and
stochastic gradient descent on quadratic functions, when the gradients are
delayed and reflect iterates from $\tau$ rounds ago. First, we show that
without stochastic noise, delays strongly affect the attainable optimization
error: In fact, the error can be as bad as non-delayed gradient descent ran on
only $1/\tau$ of the gradients. In sharp contrast, we quantify how stochastic
noise makes the effect of delays negligible, improving on previous work which
only showed this phenomenon asymptotically or for much smaller delays. Also, in
the context of distributed optimization, the results indicate that the
performance of gradient descent with delays is competitive with synchronous
approaches such as mini-batching. Our results are based on a novel technique
for analyzing convergence of optimization algorithms using generating
functions.
