  Distributed data-parallel training has been widely adopted for deep neural
network (DNN) models. Although current deep learning (DL) frameworks scale well
for dense models like image classification models, we find that these DL
frameworks have relatively low scalability for sparse models like natural
language processing (NLP) models that have highly sparse embedding tables. Most
existing works overlook the sparsity of model parameters thus suffering from
significant but unnecessary communication overhead. In this paper, we propose
EmbRace, an efficient communication framework to accelerate communications of
distributed training for sparse models. EmbRace introduces Sparsity-aware
Hybrid Communication, which integrates AlltoAll and model parallelism into
data-parallel training, so as to reduce the communication overhead of highly
sparse parameters. To effectively overlap sparse communication with both
backward and forward computation, EmbRace further designs a 2D Communication
Scheduling approach which optimizes the model computation procedure, relaxes
the dependency of embeddings, and schedules the sparse communications of each
embedding row with a priority queue. We have implemented a prototype of EmbRace
based on PyTorch and Horovod, and conducted comprehensive evaluations with four
representative NLP models. Experimental results show that EmbRace achieves up
to 2.41X speedup compared to the state-of-the-art distributed training
baselines.
