  Modern deep learning models are often trained in parallel over a collection
of distributed machines to reduce training time. In such settings,
communication of model updates among machines becomes a significant performance
bottleneck and various lossy update compression techniques have been proposed
to alleviate this problem. In this work, we introduce a new, simple yet
theoretically and practically effective compression technique: natural
compression (NC). Our technique is applied individually to all entries of the
to-be-compressed update vector and works by randomized rounding to the nearest
(negative or positive) power of two, which can be computed in a "natural" way
by ignoring the mantissa. We show that compared to no compression, NC increases
the second moment of the compressed vector by not more than the tiny factor
$\frac{9}{8}$, which means that the effect of NC on the convergence speed of
popular training algorithms, such as distributed SGD, is negligible. However,
the communications savings enabled by NC are substantial, leading to
$3$-$4\times$ improvement in overall theoretical running time. For applications
requiring more aggressive compression, we generalize NC to natural dithering,
which we prove is exponentially better than the common random dithering
technique. Our compression operators can be used on their own or in combination
with existing operators for a more aggressive combined effect and offer new
state-of-the-art both in theory and practice.
