  A rich body of prior work has highlighted the existence of communication
bottlenecks in synchronous data-parallel training. To alleviate these
bottlenecks, a long line of recent work proposes gradient and model compression
methods. In this work, we evaluate the efficacy of gradient compression methods
and compare their scalability with optimized implementations of synchronous
data-parallel SGD across more than 200 different setups. Surprisingly, we
observe that only in 6 cases out of more than 200, gradient compression methods
provide speedup over optimized synchronous data-parallel training in the
typical data-center setting. We conduct an extensive investigation to identify
the root causes of this phenomenon, and offer a performance model that can be
used to identify the benefits of gradient compression for a variety of system
setups. Based on our analysis, we propose a list of desirable properties that
gradient compression methods should satisfy, in order for them to provide a
meaningful end-to-end speedup.
