  It has been reported that the communication cost for synchronizing gradients
can be a bottleneck, which limits the scalability of distributed deep learning.
Using low-precision gradients is a promising technique for reducing the
bandwidth requirement. In this work, we propose Auto Precision Scaling (APS),
an algorithm that can improve the accuracy when we communicate gradients by
low-precision floating-point values. APS can improve the accuracy for all
precisions with a trivial communication cost. Our experimental results show
that for many applications, APS can train state-of-the-art models by 8-bit
gradients with no or only a tiny accuracy loss (<0.05%). Furthermore, we can
avoid any accuracy loss by designing a hybrid-precision technique. Finally, we
propose a performance model to evaluate the proposed method. Our experimental
results show that APS can get a significant speedup over state-of-the-art
methods. To make it available to researchers and developers, we design and
implement CPD (Customized-Precision Deep Learning) system, which can simulate
the training process using an arbitrary low-precision customized floating-point
format. We integrate CPD into PyTorch and make it open-source.
