  With the rapid growth of data, distributed momentum stochastic gradient
descent~(DMSGD) has been widely used in distributed learning, especially for
training large-scale deep models. Due to the latency and limited bandwidth of
the network, communication has become the bottleneck of distributed learning.
Communication compression with sparsified gradient, abbreviated as \emph{sparse
communication}, has been widely employed to reduce communication cost. All
existing works about sparse communication in DMSGD employ local momentum, in
which the momentum only accumulates stochastic gradients computed by each
worker locally. In this paper, we propose a novel method, called
\emph{\underline{g}}lobal \emph{\underline{m}}omentum
\emph{\underline{c}}ompression~(GMC), for sparse communication. Different from
existing works that utilize local momentum, GMC utilizes global momentum.
Furthermore, to enhance the convergence performance when using more aggressive
sparsification compressors (e.g., RBGS), we extend GMC to GMC+. We
theoretically prove the convergence of GMC and GMC+. To the best of our
knowledge, this is the first work that introduces global momentum for sparse
communication in distributed learning. Empirical results demonstrate that,
compared with the local momentum counterparts, our GMC and GMC+ can achieve
higher test accuracy and exhibit faster convergence, especially under non-IID
data distribution.
