  Diffusion models have emerged as dominant performers for image generation. To
support training large diffusion models, this paper studies pipeline parallel
training of diffusion models and proposes DiffusionPipe, a synchronous pipeline
training system that advocates innovative pipeline bubble filling technique,
catering to structural characteristics of diffusion models. State-of-the-art
diffusion models typically include trainable (the backbone) and non-trainable
(e.g., frozen input encoders) parts. We first unify optimal stage partitioning
and pipeline scheduling of single and multiple backbones in representative
diffusion models with a dynamic programming approach. We then propose to fill
the computation of non-trainable model parts into idle periods of the pipeline
training of the backbones by an efficient greedy algorithm, thus achieving high
training throughput. Extensive experiments show that DiffusionPipe can achieve
up to 1.41x speedup over pipeline parallel methods and 1.28x speedup over data
parallel training on popular diffusion models.
