  Gradient aggregation has long been identified as a major bottleneck in
today's large-scale distributed machine learning training systems. One
promising solution to mitigate such bottlenecks is gradient compression,
directly reducing communicated gradient data volume. However, in practice, many
gradient compression schemes do not achieve acceleration of the training
process while also preserving accuracy.
  In this work, we identify several common issues in previous gradient
compression systems and evaluation methods. These issues include excessive
computational overheads; incompatibility with all-reduce; and inappropriate
evaluation metrics, such as not using an end-to-end metric or using a 32-bit
baseline instead of a 16-bit baseline. We propose several general design and
evaluation techniques to address these issues and provide guidelines for future
work. Our preliminary evaluation shows that our techniques enhance the system's
performance and provide a clearer understanding of the end-to-end utility of
gradient compression methods.
