  Stochastic Gradient Descent (SGD) has become one of the most popular
optimization methods for training machine learning models on massive datasets.
However, SGD suffers from two main drawbacks: (i) The noisy gradient updates
have high variance, which slows down convergence as the iterates approach the
optimum, and (ii) SGD scales poorly in distributed settings, typically
experiencing rapidly decreasing marginal benefits as the number of workers
increases. In this paper, we propose a highly parallel method, CentralVR, that
uses error corrections to reduce the variance of SGD gradient updates, and
scales linearly with the number of worker nodes. CentralVR enjoys low iteration
complexity, provably linear convergence rates, and exhibits linear performance
gains up to hundreds of cores for massive datasets. We compare CentralVR to
state-of-the-art parallel stochastic optimization methods on a variety of
models and datasets, and find that our proposed methods exhibit stronger
scaling than other SGD variants.
