  Running Stochastic Gradient Descent (SGD) in a decentralized fashion has
shown promising results. In this paper we propose Moniqua, a technique that
allows decentralized SGD to use quantized communication. We prove in theory
that Moniqua communicates a provably bounded number of bits per iteration,
while converging at the same asymptotic rate as the original algorithm does
with full-precision communication. Moniqua improves upon prior works in that it
(1) requires zero additional memory, (2) works with 1-bit quantization, and (3)
is applicable to a variety of decentralized algorithms. We demonstrate
empirically that Moniqua converges faster with respect to wall clock time than
other quantized decentralized algorithms. We also show that Moniqua is robust
to very low bit-budgets, allowing 1-bit-per-parameter communication without
compromising validation accuracy when training ResNet20 and ResNet110 on
CIFAR10.
