  This paper introduces PipeFusion, a novel approach that harnesses multi-GPU
parallelism to address the high computational and latency challenges of
generating high-resolution images with diffusion transformers (DiT) models.
PipeFusion splits images into patches and distributes the network layers across
multiple devices. It employs a pipeline parallel manner to orchestrate
communication and computations. By leveraging the high similarity between the
input from adjacent diffusion steps, PipeFusion eliminates the waiting time in
the pipeline by reusing the one-step stale feature maps to provide context for
the current step. Our experiments demonstrate that it can generate higher image
resolution where existing DiT parallel approaches meet OOM. PipeFusion
significantly reduces the required communication bandwidth, enabling DiT
inference to be hosted on GPUs connected via PCIe rather than the more costly
NVLink infrastructure, which substantially lowers the overall operational
expenses for serving DiT models. Our code is publicly available at
https://github.com/PipeFusion/PipeFusion.
