  Pipeline parallelism has been demonstrated to be a remarkable approach to
improve throughput for training deep neural networks with billions of
parameters over heterogeneous clusters. The 1F1B scheduling plan is a widely
adopted strategy for memory and performance optimization, which interchanges
the forward and backward stage computations of different micro-batches. On the
other hand, a common issue in using the 1F1B scheduling is that stage
computation is delayed due to the data transfer when network resources are
preempted by other tasks, even with the minimum communication between stages.
The exclusive access of these network resources cannot be guaranteed in cloud
offerings. We present a general scheduling technique to accommodate pipeline
parallelism to preempted network environments at the expense of a certain
amount of memory pressure. The core concept is to extend 1F1B schedule scheme
to kFkB, which groups k micro-batches, and alternately executes k forward and
backward computations. We propose Ada-Grouper, an adaptive kFkB scheduler which
regularly adjusts the number of group members k to maintain an optimal balance
between communication and computation efficiency correspond to changes in a
changing network environment under the memory limit. Experimental results
demonstrate that our design maintain stable performance for pipeline
parallelism, yielding a performance increase of up from 4% to 30%, compared
with 1F1B in preempted network scenarios.
