  Distributed deep learning (DL) has become prevalent in recent years to reduce
training time by leveraging multiple computing devices (e.g., GPUs/TPUs) due to
larger models and datasets. However, system scalability is limited by
communication becoming the performance bottleneck. Addressing this
communication issue has become a prominent research topic. In this paper, we
provide a comprehensive survey of the communication-efficient distributed
training algorithms, focusing on both system-level and algorithmic-level
optimizations. We first propose a taxonomy of data-parallel distributed
training algorithms that incorporates four primary dimensions: communication
synchronization, system architectures, compression techniques, and parallelism
of communication and computing tasks. We then investigate state-of-the-art
studies that address problems in these four dimensions. We also compare the
convergence rates of different algorithms to understand their convergence
speed. Additionally, we conduct extensive experiments to empirically compare
the convergence performance of various mainstream distributed training
algorithms. Based on our system-level communication cost analysis, theoretical
and experimental convergence speed comparison, we provide readers with an
understanding of which algorithms are more efficient under specific distributed
environments. Our research also extrapolates potential directions for further
optimizations.
