  We consider the setting of distributed empirical risk minimization where
multiple machines compute the gradients in parallel and a centralized server
updates the model parameters. In order to reduce the number of communications
required to reach a given accuracy, we propose a \emph{preconditioned}
accelerated gradient method where the preconditioning is done by solving a
local optimization problem over a subsampled dataset at the server. The
convergence rate of the method depends on the square root of the relative
condition number between the global and local loss functions. We estimate the
relative condition number for linear prediction models by studying
\emph{uniform} concentration of the Hessians over a bounded domain, which
allows us to derive improved convergence rates for existing preconditioned
gradient methods and our accelerated method. Experiments on real-world datasets
illustrate the benefits of acceleration in the ill-conditioned regime.
