  Classical stochastic gradient methods for optimization rely on noisy gradient
approximations that become progressively less accurate as iterates approach a
solution. The large noise and small signal in the resulting gradients makes it
difficult to use them for adaptive stepsize selection and automatic stopping.
We propose alternative "big batch" SGD schemes that adaptively grow the batch
size over time to maintain a nearly constant signal-to-noise ratio in the
gradient approximation. The resulting methods have similar convergence rates to
classical SGD, and do not require convexity of the objective. The high fidelity
gradients enable automated learning rate selection and do not require stepsize
decay. Big batch methods are thus easily automated and can run with little or
no oversight.
