  We study the trade-off between convergence rate and sensitivity to stochastic
additive gradient noise for first-order optimization methods. Ordinary Gradient
Descent (GD) can be made fast-and-sensitive or slow-and-robust by increasing or
decreasing the stepsize, respectively. However, it is not clear how such a
trade-off can be navigated when working with accelerated methods such as
Polyak's Heavy Ball (HB) or Nesterov's Fast Gradient (FG) methods. We consider
three classes of functions: (1) smooth strongly convex quadratics, (2) smooth
strongly convex functions, and (3) functions that satisfy the
Polyak-Lojasiewicz property and have one-sided Lipschitz gradients. For each
function class, we present a tractable way to compute the convergence rate and
sensitivity to additive gradient noise for a broad family of first-order
methods, and we present algorithm designs that trade off these competing
performance metrics. Each design consists of a simple analytic update rule with
two states of memory, similar to HB and FG. Moreover, each design has a scalar
tuning parameter that explicitly trades off convergence rate and sensitivity to
additive gradient noise. We numerically validate the performance of our designs
by comparing their convergence rate and sensitivity to those of many other
algorithms, and through simulations on Nesterov's "bad function".
