  To train modern large DNN models, pipeline parallelism has recently emerged,
which distributes the model across GPUs and enables different devices to
process different microbatches in pipeline. Earlier pipeline designs allow
multiple versions of model parameters to co-exist (similar to asynchronous
training), and cannot ensure the same model convergence and accuracy
performance as without pipelining. Synchronous pipelining has recently been
proposed which ensures model performance by enforcing a synchronization barrier
between training iterations. Nonetheless, the synchronization barrier requires
waiting for gradient aggregation from all microbatches and thus delays the
training progress. Optimized pipeline planning is needed to minimize such wait
and hence the training time, which has not been well studied in the literature.
This paper designs efficient, near-optimal algorithms for expediting
synchronous pipeline-parallel training of modern large DNNs over arbitrary
inter-GPU connectivity. Our algorithm framework comprises two components: a
pipeline partition and device mapping algorithm, and a pipeline scheduler that
decides processing order of microbatches over the partitions, which together
minimize the per-iteration training time. We conduct thorough theoretical
analysis, extensive testbed experiments and trace-driven simulation, and
demonstrate our scheme can accelerate training up to 157% compared with
state-of-the-art designs.
