  Data-parallel distributed training of deep neural networks (DNN) has gained
very widespread adoption, but can still experience communication bottlenecks.
To address this issue, entire families of compression mechanisms have been
developed, including quantization, sparsification, and low-rank approximation,
some of which are seeing significant practical adoption. Despite this progress,
almost all known compression schemes apply compression uniformly across DNN
layers, although layers are heterogeneous in terms of parameter count and their
impact on model accuracy. In this work, we provide a general framework for
adapting the degree of compression across the model's layers dynamically during
training, improving the overall compression, while leading to substantial
speedups, without sacrificing accuracy. Our framework, called L-GreCo, is based
on an adaptive algorithm, which automatically picks the optimal compression
parameters for model layers guaranteeing the best compression ratio while
satisfying an error constraint. Extensive experiments over image classification
and language modeling tasks shows that L-GreCo is effective across all existing
families of compression methods, and achieves up to 2.5$\times$ training
speedup and up to 5$\times$ compression improvement over efficient
implementations of existing approaches, while recovering full accuracy.
Moreover, L-GreCo is complementary to existing adaptive algorithms, improving
their compression ratio by 50% and practical throughput by 66%.
