  For SGD based distributed stochastic optimization, computation complexity,
measured by the convergence rate in terms of the number of stochastic gradient
calls, and communication complexity, measured by the number of inter-node
communication rounds, are two most important performance metrics. The classical
data-parallel implementation of SGD over $N$ workers can achieve linear speedup
of its convergence rate but incurs an inter-node communication round at each
batch. We study the benefit of using dynamically increasing batch sizes in
parallel SGD for stochastic non-convex optimization by charactering the
attained convergence rate and the required number of communication rounds. We
show that for stochastic non-convex optimization under the P-L condition, the
classical data-parallel SGD with exponentially increasing batch sizes can
achieve the fastest known $O(1/(NT))$ convergence with linear speedup using
only $\log(T)$ communication rounds. For general stochastic non-convex
optimization, we propose a Catalyst-like algorithm to achieve the fastest known
$O(1/\sqrt{NT})$ convergence with only $O(\sqrt{NT}\log(\frac{T}{N}))$
communication rounds.
