  Nowadays, large and complex deep learning (DL) models are increasingly
trained in a distributed manner across multiple worker machines, in which
extensive communications between workers pose serious scaling problems. In this
article, we present a quantitative survey of communication optimization
techniques for data parallel distributed DL. We first identify the major
communication challenges and classify the existing solutions into three levels,
namely the learning algorithm, the system architecture, and the network
infrastructure. We present the state-of-the-art communication optimization
techniques and conduct a comparative study of seven common lossless distributed
DL methods on a 32-GPU cluster with 100Gbps InfiniBand (IB). We show that (1)
the DL models with low model intensity (such as BERT and BERT-Large) are
difficult to scale out even with the best available lossless algorithm over
100Gbps IB; (2) the system architecture and scheduling algorithms have a
critical impact on the scaling property. We conclude the article with
discussions on the open issues for further investigations.
