  The employment of high-performance servers and GPU accelerators for training
deep neural network models have greatly accelerated recent advances in deep
learning (DL). DL frameworks, such as TensorFlow, MXNet, and Caffe2, have
emerged to assist DL researchers to train their models in a distributed manner.
Although current DL frameworks scale well for image classification models,
there remain opportunities for scalable distributed training on natural
language processing (NLP) models. We found that current frameworks show
relatively low scalability on training NLP models due to the lack of
consideration to the difference in sparsity of model parameters. In this paper,
we propose Parallax, a framework that optimizes data parallel training by
utilizing the sparsity of model parameters. Parallax introduces a hybrid
approach that combines Parameter Server and AllReduce architectures to optimize
the amount of data transfer according to the sparsity. Experiments show that
Parallax built atop TensorFlow achieves scalable training throughput on both
dense and sparse models while requiring little effort from its users. Parallax
achieves up to 2.8x, 6.02x speedup for NLP models than TensorFlow and Horovod
with 48 GPUs, respectively. The training speed for the image classification
models is equal to Horovod and 1.53x faster than TensorFlow.
