  Gradient-based optimization methods implemented on distributed computing
architectures are increasingly used to tackle large-scale machine learning
applications. A key bottleneck in such distributed systems is the high
communication overhead for exchanging information, such as stochastic
gradients, between workers. The inherent causes of this bottleneck are the
frequent communication rounds and the full model gradient transmission in every
round. In this study, we present SASG, a communication-efficient distributed
algorithm that enjoys the advantages of sparse communication and adaptive
aggregated stochastic gradients. By dynamically determining the workers who
need to communicate through an adaptive aggregation rule and sparsifying the
transmitted information, the SASG algorithm reduces both the overhead of
communication rounds and the number of communication bits in the distributed
system. For the theoretical analysis, we introduce an important auxiliary
variable and define a new Lyapunov function to prove that the
communication-efficient algorithm is convergent. The convergence result is
identical to the sublinear rate of stochastic gradient descent, and our result
also reveals that SASG scales well with the number of distributed workers.
Finally, experiments on training deep neural networks demonstrate that the
proposed algorithm can significantly reduce communication overhead compared to
previous methods.
