  Distributed deep learning is becoming increasingly popular due to the
expanding demand for computing resources for deep learning models with a larger
amount of parameters. Different from traditional training approaches,
data-parallel training allows multiple compute nodes to train large deep
learning models simultaneously in order to boost the training efficiency. In
this paper, we present and compare six strategies for data-parallel training
using PyTorch on the language model GPT-2 with 100M parameters using a
qualitative approach. These strategies are Single GPU, Single Parameter Server,
Distributed Parameter Server, Horovod, Distributed Parameter Server with Apex
mixed-precision strategy, and Horovod with Apex mixed-precision strategy. We
also analyze the quantitative experiment results from each strategy. In the
end, we draw the conclusion that the Distributed Parameter Server with Apex
mixedprecision strategy has the best performance on single node training, while
Horovod with Apex is the most robust approach to use when we have single or
multiple nodes.
