  The size of deep neural networks (DNNs) grows rapidly as the complexity of
the machine learning algorithm increases. To satisfy the requirement of
computation and memory of DNN training, distributed deep learning based on
model parallelism has been widely recognized. We propose a new pipeline
parallelism training framework, BaPipe, which can automatically explore
pipeline parallelism training methods and balanced partition strategies for DNN
distributed training. In BaPipe, each accelerator calculates the forward
propagation and backward propagation of different parts of networks to
implement the intra-batch pipeline parallelism strategy. BaPipe uses a new load
balancing automatic exploration strategy that considers the parameters of DNN
models and the computation, memory, and communication resources of accelerator
clusters. We have trained different DNNs such as VGG-16, ResNet-50, and GNMT on
GPU clusters and simulated the performance of different FPGA clusters. Compared
with state-of-the-art data parallelism and pipeline parallelism frameworks,
BaPipe provides up to 3.2x speedup and 4x memory reduction in various
platforms.
