  Compressed communication, in the form of sparsification or quantization of
stochastic gradients, is employed to reduce communication costs in distributed
data-parallel training of deep neural networks. However, there exists a
discrepancy between theory and practice: while theoretical analysis of most
existing compression methods assumes compression is applied to the gradients of
the entire model, many practical implementations operate individually on the
gradients of each layer of the model. In this paper, we prove that layer-wise
compression is, in theory, better, because the convergence rate is upper
bounded by that of entire-model compression for a wide range of biased and
unbiased compression methods. However, despite the theoretical bound, our
experimental study of six well-known methods shows that convergence, in
practice, may or may not be better, depending on the actual trained model and
compression ratio. Our findings suggest that it would be advantageous for deep
learning frameworks to include support for both layer-wise and entire-model
compression.
