  Synchronous stochastic gradient descent (SGD) is the most common method used
for distributed training of deep learning models. In this algorithm, each
worker shares its local gradients with others and updates the parameters using
the average gradients of all workers. Although distributed training reduces the
computation time, the communication overhead associated with the gradient
exchange forms a scalability bottleneck for the algorithm. There are many
compression techniques proposed to reduce the number of gradients that needs to
be communicated. However, compressing the gradients introduces yet another
overhead to the problem. In this work, we study several compression schemes and
identify how three key parameters affect the performance. We also provide a set
of insights on how to increase performance and introduce a simple
sparsification scheme, random-block sparsification, that reduces communication
while keeping the performance close to standard SGD.
